Combine CSV Files

JasonW

This example uses the following open event of a scripted data set to implement this feature:

importPackage(Packages.java.io);
directory = new File(params["fileDirectory"]); 
files = directory.listFiles();
csvfiles = [];
for( i=0; i<files.length; i++ ){
 if( files[i].toString().endsWith(".csv") && files[i].toString().contains("mydata") ){
  csvfiles.push(files[i].getAbsolutePath());
 }
} 
idx=0;
mx = csvfiles.length;
importPackage(Packages.java.net);
importPackage(Packages.java.io);
inf = new FileInputStream(csvfiles[0]);
inp = new BufferedReader(new InputStreamReader(inf))
inputLine = inp.readLine(); //throw away header
inputLine = inp.readLine(); //throw away header
The open method is looking for mydata*.csv files and expects each file to have two rows of header information.
The fetch event handler looks like:
importPackage(Packages.java.lang);
function openNewFile(){
  
  inf = new FileInputStream(csvfiles[idx]);
  inp = new BufferedReader(new InputStreamReader(inf))
  inputLine = inp.readLine(); //throw away header
  inputLine = inp.readLine(); //throw away header
  inputLine = inp.readLine(); //get first file
  //if empty csv is encountered go to next file
  if( inputLine == null ){
   inp.close();
   idx++
   openNewFile();
  }
}
var inputLine;
if ((inputLine = inp.readLine()) == null){
 idx++;
 if( idx >= mx ){
  inp.close();
  return false;
 }else{
  openNewFile();
 }
}
var myrowarray =inputLine.split("\,");
row["col1"] = myrowarray[0];
row["col2"] = myrowarray[1];
row["col3"] = myrowarray[2];

<br />
This event just reads the file and if it is at the end of the file it opens the next file. Note that there is very little error checking.