CSV Import / Export: bug + PICK field
As you know, importing to Pods is a subject that is near and dear to my heart. I've been playing with this in my free time, and found some bugs / solutions.
1) In PodAPI.class.php, the function csv_to_php() does not handle CSV files correctly. I found that when the first data column is enclosed in quotes, it trips up this function. I've tried several other CSV parsing functions on php.net, and none of them worked.
What does work is importing a csv via fgetcsv(). I wrote this code, and it works: <pre> <?php $php_data = pods_csv_import('my_file.csv', ',', 5000); $Record = new PodAPI($use_pod, 'php'); $Record->import($php_data);
function pods_csv_import($filename, $delimiter = ',', $max_line_len = 1000) { $first_row = true; if (false == ($handle = fopen($filename, "r"))) die("<p>Could not open $filename</p>"); while (($data = fgetcsv($handle, $max_line_len, $delimiter)) !== FALSE) { if ($first_row) { $field_names = $data; $first_row = false; } else { $tmp[$field] = $data[$key]; } $out[] = $tmp; } } fclose($handle); return $out; } ?> </pre>
2) I've been looking for a way to import / export pick columns via CSV. The CSV standard doesn't have a protocol for storing nested data, so I've come up with my own.
My steps are: + Replace any commas in the PICK array with &#44; + Implode the array into a comma delimited string + Store the string as 'array(' . $string . ')'
Again, I've tested this, and it works exporting a Pod then importing it too a new Pod. My test Pod has 34 columns (including 5 desc and 8 PICK). The code for export is: <pre> <?php $Record = new PodAPI('my_pod', 'csv'); $data = $Record->export();
if (($fp = fopen('file.csv', 'w')) === FALSE) { //Attempt to open 'file.csv' for writing and check for success. die ("Could not open $out_file. Export aborted."); //Fail if unable to. }
$data_keys = array_keys($data[0]); //Get the pod column labels fputcsv($fp, $data_keys); //Write the column labels as the first line of the CSV file. foreach ($data as $line) { //Loop through the data line by line foreach ($line as &$field) { //Loop through each line of data by field if (is_array($field)) { //If the field is a PICK column array_walk($field, 'comma_trans'); //Translate any commas in the field to HTML "&#44;" $field = 'array(' . implode(', ', $field) . ')'; //Implode the items into a comma separated list wrapped in array(). } } fputcsv($fp, $line); //Write the line to the file. }
function comma_trans(&$item){ $item = strtr($item, ',', '&#44'); //Translates any commas into their HTML entity. }
?> </pre>
The import code gets added to the above code just after "} else {": <pre> foreach ($field_names as $key => $field) { if ($data[$key] <> ($new_data = preg_replace('/^array((.*))$/is', '$1', $data[$key]))) { $new_data = explode(",",trim($new_data)); array_walk($new_data, 'comma_trans'); $data[$key] = $new_data; } </pre>
and the comma_trans function gets added with this modification: <pre> function comma_trans(&$item){ $item = strtr($item, '&#44', ','); //Translates HTML &#44; to commas } </pre>
I've added packages to the library with the full code of my "Import" and "Export" pages.
10 Answers
Wow...! Thanks for your contributions!
The script in the extend center did install for me so I used this export code, but when exporting a text field i'm getting a weird problem. For every "," in the text it adds row and a new collum.
see: http://94.23.148.25/file.csv
Using an export, is there any way to pull the guid of uploaded files?
@magi182 - The exporter will soon be updated to support the new file columns.
Please also fixes if a text field got -> , <- then it don't auto create new rows/columns
look at my example csv file what the problem is if got
@marcelp: I looked at your data example above. In Excel and OpenOffice, your data comes in as a valid CSV file, with all the rows and columns matched. I didn't try this in Pods, but I've brought in data sets like this without a problem using the Import function I added to "Extend": http://pods.uproot.us/packages/pod-page-csv-import/
Is this how you are bringing the data back in? The default Pods Api Import has some problems in with it recognizing complex CSV files.
@magi182, @logikal16: I'll be honest, some paying projects have gotten in the way of my Pods development. At the moment, I'm not running anything later than 1.7.6 on my sites. Hopefully this week's holiday will give me some time to upgrade my sites and these plugins.
I was getting some out of Memory issues, even though I have control over the php.ini file.
Turns out somewhere in the pods module the memory limit is being reset to 64MB, something that doesn't work for importing 200 000+ reccords.
If I may, I would suggest 3 different suggestions: 1) removing the setting of a 64MB limit so people don't have to try to find a place to put their own in the code instead of fixing it in php.ini 2) making the limit you do put in higher if you choose to keep it in 3)Making the code more efficient. (there is something non-optimal if 64Mb ram is not enough to process a 4mb csv file)
thank you all for the great work on this.
I have copied your post to the roadmap for discussion in the next dev meeting. I'll keep you posted.
@steph - While there is plenty of room for the API to become more efficient, setting the memory limit is not the job of Pods. You can add it to your PHP file, when necessary, by placing the following before your API code.
<pre> ini_set('memory_limit', '64M'); </pre>


