I'm trying to create excel sheets using coldfusion's spreadsheetaddrows function. Because the tables I'm creating spreadsheets from can be very large, I'm creating the excel file in chunks to avoid timing out the server. Everything works fine at first, but as the process goes on it really slows down. For example, I was testing with a table that had 50,000 plus rows and almost 100 columns. Five hours later, the excel file hadn't even reached 10,000 rows. =T
It seems that spreadsheetaddrows is the problem according to google. Is there a faster and better way for me to do this?
<cfloop index="i" from="#start#" to="#end#" step="10">
<!-- variables -->
<cfset plus = #i# + 9>
<cfif #plus# gt #end#>
<cfset plus = #end#>
</cfif>
<!-- /variables -->
<!-- get i to i+9 records -->
<cfquery name="query" datasource="datasource">
select *
from ( select a.*, rownum rnum
from ( select * from table order by id) a
where rownum <= #plus#
)
where rnum >= #i#
</cfquery>
<cfif it's the first time doing this>
<!-- import from database query and save as excel sheet -->
<cfset spreadsheetAddRows(theSheet, query)>
<cfif reached the end of the query>
<cfspreadsheet action = "write"
overwrite = "true"
filename = "file.xls"
name="theSheet"
>
</cfif>
<cfelseif 2nd or more time doing this>
<!-- add new rows to excel object -->
<cfset spreadsheetAddRows(temp, query)>
<!-- overwrite existing xls file with new data -->
<cfspreadsheet action="write"
overwrite="true"
filename="file.xls"
name="temp"
>
</cfif>
I was curious so I tried another approach. The code below executes successfully and quickly with a small dataset. For what you are doing, this might still be slow and it might create java heap space problems, but it's probably worth a shot.
First - Get your query results into a spreadsheet.
<cfquery name="dbdata" datasource="dw">
select query goes here
</cfquery>
<cfspreadsheet action="write" overwrite="yes" query="dbdata" filename="#completeFileName#">
Then read that spreadsheet, do what you have to do, and update the file.
<cfscript>
abc = spreadsheetread(completefilename);
spreadsheetsetcellvalue(abc, 'new value', 2, 1);
spreadsheetwrite(abc, completeFileName, true);
</cfscript>
When I opened the file in excel, row 1 had column headers from the query results, and cell A2 contained 'new value'.
I have excel file that can contain commas in some fields. This can cause problem if I want to use cfspreadsheet to convert my file to csv. I was wondering if there is the way to replace or convert all commas with the \. After I replace all commas then I will be able to use cfspreadsheet to create csv. Here is my code how I read my file:
<cfspreadsheet action = "read" format="csv" src="filePath\myFile.xlsx" name="csvvar">
If anyone can help with this problem please let me know. Thank you.
Convert from Excel to query. Then, in the cell data of each row, replace "," by "\". Something like this
<cfspreadsheet
action = "read"
src="filePath\myFile.xlsx"
query="excelquery"
sheet="1">
<!--- Create CSV file in current directory--->
<cffile action="write" file="#expandpath('result.csv')#" output="">
<cfset columns = arraynew(1)>
<!--- Store the list of column names as an array --->
<cfset columns = listToArray(excelquery.ColumnList)>
<cfoutput query="excelquery">
<cfset rowList = "">
<cfloop from="1" to="#arraylen(columns)#" index="n">
<cfset colName = columns[n]>
<cfset cellData = evaluate("#colName#[currentrow]")>
<!--- Replace , by \ in each cell --->
<cfset cellData = replace(cellData, ",", "\", "all")>
<!--- Comma-separated row data --->
<cfset rowList = listAppend(rowList,cellData)>
</cfloop>
<!--- Place a carriage-return at the end of the row --->
<cfset rowList = rowList & '<br>'>
<!--- Append row to CSV file --->
<cffile action="append" file="#expandpath('result.csv')#" output="#rowList#" >
</cfoutput>
I have a request from a client to generate an Excel spreadsheet from a query. I have the query kicking out the fields and I can generate the Excel file without a hitch. The problem comes when the client takes that Excel file and then tries to manipulate it.
The majority of the trouble comes from fields that should be marked as currency or dates. I am, with some struggle, able to generate a "real" date field. Before this Excel was not sorting the dates properly. I was able to call an Excel formula by using the code below. DateValue forces Excel to acknowledge this as a real date field. However, this fails when this file is manipulated through Excel.
<cfset SpreadsheetSetCellFormula(s
,"DATEVALUE(#Chr(34)##Replacement_ETD##Chr(34)#)"
, therow
, 9)>
The next problem is the currency field. I can't get Excel to acknowledge the values as a currency. It always comes up custom. When this is set, the SUM function won't work in Excel. You can add the fields individually like A1+B1+C1 = TOTAL. However, this won't be helpful when there are 200 rows.
I was able to get a suggestion from another CF programmer who had a similar situation. He generated the Excel file first with the proper headings and set the columns to their proper fields such as date and currency, etc.
The next step would be to fill in the fields row by row and they should be properly formatted.
Code:
<cfset filename = expandPath("./reports/arrivals.xlsx")>
<cfspreadsheet action="read" src = "#filename#" name = "s" >
<cfset therow = 0>
<cfoutput query="myExcel" startrow="1">
<cfset therow = myExcel.currentrow + 1>
<cfset SpreadsheetSetCellValue(s, Incumbent, therow, 1)>
<cfset SpreadsheetSetCellValue(s, Section, therow, 2)>
<cfset SpreadsheetSetCellValue(s, Position_Number, therow, 3)>
<cfset SpreadsheetSetCellValue(s, Position_Title, therow, 4)>
<cfset SpreadsheetSetCellValue(s, Incumbent_Emplyment_Type, therow, 5)>
<cfset SpreadsheetSetCellValue(s, Incumbent_ETD, therow, 6)>
<cfset SpreadsheetSetCellValue(s, Tour_Comments, therow, 7)>
<cfset SpreadsheetSetCellValue(s, Replacement, therow, 8)>
<cfset SpreadsheetSetCellValue(s, Replacement_ETA, therow, 9)>
</cfoutput>
<cfheader name="content-disposition" value="attachment; filename=Departures_(#DateFormat(now(),'mmddyy')#).xls">
<cfcontent type="application/msexcel" variable="#spreadsheetReadBinary(s)#" reset="true">
The data in the cells has already been properly formatted. When this file is generated and streamed to the user the columns are not formatted as expected.
Does anyone else know if this method will work or have a better suggestion on getting CF to generate a proper date and currency field for Excel to acknowledge?
Adobe ColdFusion v10 running on RHEL 5.
Per request here is some code using queryNew that will generate code dates and currency.
Step one: I created an Excel file with the first row frozen and it has the column header. Column one has been designated as the date the format is long date - mm/dd/yyy; Column two is Dollar which as been set to currency.
I read that file then fill in the rows and stream the file to the user for download.
<cfset filename = expandPath("./reports/Test.xlsx")>
<cfspreadsheet action="read" src = "#filename#" name = "s" >
<cfset myQuery = QueryNew("MyDate, Dollar", "Date, Decimal")>
<cfset newRow = QueryAddRow(MyQuery, 5)>
<cfset temp = QuerySetCell(myQuery, "MyDate", "03-11-2000", 1)>
<cfset temp = QuerySetCell(myQuery, "Dollar", "403.45", 1)>
<cfset temp = QuerySetCell(myQuery, "MyDate", "01-01-2009", 2)>
<cfset temp = QuerySetCell(myQuery, "Dollar", "603.22", 2)>
<cfset temp = QuerySetCell(myQuery, "MyDate", "09-21-2013", 3)>
<cfset temp = QuerySetCell(myQuery, "Dollar", "103.55", 3)>
<cfset temp = QuerySetCell(myQuery, "MyDate", "01-15-2005", 4)>
<cfset temp = QuerySetCell(myQuery, "Dollar", "3.33", 4)>
<cfset temp = QuerySetCell(myQuery, "MyDate", "07-22-2003", 5)>
<cfset temp = QuerySetCell(myQuery, "Dollar", "13.75", 5)>
<cfset therow = 0>
<cfoutput query="myQuery" startrow="1">
<cfset therow = myQuery.currentrow + 1>
<cfset SpreadsheetSetCellValue(s, DateFormat(MyDate, 'mm/dd/yyyy'), therow, 1)>
<cfset SpreadsheetSetCellValue(s, Dollar, therow, 2)>
#myQuery.currentrow# <br>
#myQuery.MyDate# <br>
#myQuery.Dollar# <br>
</cfoutput>
<cfheader name="content-disposition" value="attachment;
filename=Departures_(#DateFormat(now(),'mmddyy')#).xls">
<cfcontent type="application/msexcel" variable="#spreadsheetReadBinary(s)#" reset="true">
You can open the file in MS Excel or in Google Sheets. Test one, with the first row frozen, we should be able to sort on the date field. My results are: The dates are not being sorted properly. On column 2 with the currency, if we try to do a SUM that does work! This had not worked before but it does now.
Also, when I try to open the file I am given the warning that this file is corrupt and Excel will try to open it. I get no such warning on Google Sheets.
CF can be a bit quirky when working with date cells. Excel is pretty good about guessing the correct cell type when a value is entered manually. However, it is a little trickier with CF. Since CF is relatively typeless, it does not always match up values and cell types correctly. Using functions that utilize a query object, instead of SpreadsheetSetCellValue(), usually produces better results. Most likely because query objects contains both values and data types. Though as of CF11, SpreadsheetSetCellValue supports a new data parameter, which allows you to specify both the value and cell data type. Since you are using CF10, try using SpreadsheetAddRows to populate the values instead.
Regarding, the warning that the file is corrupt, it is caused by the fact that the actual file content and the file extension in the download code do not match. The code is reading in an .xlsx file, but the download claims it is an .xls (application/msexcel) file. To get rid of the error, make sure the two match.
Here is a working example tested with CF11
<!---
Test.xlsx contains two columns, with headers on row 1
- Column A format: *m/d/yyyy
- Column B format: number with 2 decimal places
--->
<cfspreadsheet action="read" src="c:/temp/Test.xlsx" name="sheet" >
<cfset myQuery = QueryNew("")>
<cfset QueryAddColumn(MyQuery, "Dollar", "Decimal", [ 403.45, 703.22, 103.55, 3.33, 13.75] )>
<cfset QueryAddColumn(MyQuery, "MyDate", "date", [ parseDateTime("2000-03-11", "yyyy-mm-dd")
, parseDateTime("2009-01-01", "yyyy-mm-dd")
, parseDateTime("2013-09-21", "yyyy-mm-dd")
, parseDateTime("2005-01-15", "yyyy-mm-dd")
, parseDateTime("2003-07-22", "yyyy-mm-dd")] ) >
<cfset spreadsheetAddRows(sheet, myQuery)>
<cfset spreadsheetFormatColumn(sheet, {dataFormat="m/d/yy"}, 1)>
<cfset spreadsheetFormatColumn(sheet, {dataFormat="##,####0.00"}, 2)>
<cfheader name="content-disposition" value="attachment; filename=Departures_(#DateFormat(now(),'mmddyy')#).xlsx">
<cfcontent type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" variable="#spreadsheetReadBinary(sheet)#" reset="true">
I'm running ColdFusion 10 locally and running into two issues with a database that was recently converted from Oracle to MSSQL.
What I'm trying to do is output data from the database into a spreadsheet using cfspreadsheet. There are 3 columns in a table and these contain date / timestamps.
Issue #1: The code worked fine prior to the conversion but after the conversion all the date / time stamps appear in scientific notation format. I've tried modifying my CFQUERY with a cast convert, but all this does is show the date / timestamp fields as 'YYYY/MM/DD' instead of my intended format which is 'MM/DD/YYYY HH:MM AM/PM.' When I double click on the cell in excel, I see the correct formatting but by default it shows it as 'YYYY/MM/DD.' Any suggestions here?
Issue #2: If the column or specific cell that is supposed to use the date / time formatting I specified is empty or null, I receive a '' is an invalid date or time string error.
Here is what I am using for the CAST/ CONVERT in my cfquery:
,CAST(CONVERT(varchar(20), GYMSTARTDATE, 22) AS datetime) AS GymStartDateTime
,CAST(CONVERT(varchar(20), GYMENDDATE, 22) AS datetime) AS GymEndDateTime
It is hard to say without seeing a data sample and your actual code. However, I tried to reproduce the issue you described with CF10 and SQL Server 2005, but could not. All of the values displayed as strings, not scientific notation.
It sounds like there is something different in your data, or environment, that we are not aware of. Please put together a small example that reproduces the issue and we can debug it further.
Test 1:
<!--- raw dates --->
<cfquery name="qry" datasource="#dsn#">
SELECT StartDate, EndDate
FROM YourTable
</cfquery>
<cfspreadsheet action="write"
filename="c:/test.xls"
query="qry"
overwrite="true" />
Result 1: (Cell Type: General)
STARTDATE ENDDATE
2013-08-20 14:19:28.907 2013-09-03 14:19:28.907
2013-08-25 14:19:30.293 2013-09-03 02:19:30.293
Test 2:
<!--- convert / datetime --->
<cfquery name="qry" datasource="#dsn#">
SELECT CAST(CONVERT(varchar(20), StartDate, 22) AS datetime) AS GymStartDateTime
, CAST(CONVERT(varchar(20), EndDate, 22) AS datetime) AS GymEndDateTime
FROM YourTable
</cfquery>
<cfspreadsheet action="write"
filename="c:/test2.xls"
query="qry"
overwrite="true" />
Result 2: (Cell Type: General)
GYMSTARTDATETIME GYMENDDATETIME
2013-08-20 14:19:28.0 2013-09-03 14:19:28.0
2013-08-25 14:19:30.0 2013-09-03 02:19:30.0
Test 3:
<!--- convert / string --->
<cfquery name="qry" datasource="#dsn#">
SELECT CONVERT(varchar(20), StartDate, 22) AS GymStartDateTime
, CONVERT(varchar(20), EndDate, 22) AS GymEndDateTime
FROM YourTable
</cfquery>
<cfspreadsheet action="write"
filename="c:/test3.xls"
query="qry"
overwrite="true" />
Result 3: (Cell Type: General)
GYMSTARTDATETIME GYMENDDATETIME
08/20/13 2:19:28 PM 09/03/13 2:19:28 PM
08/25/13 2:19:30 PM 09/03/13 2:19:30 AM
If you ever figure it out, please let me know. Here's a link to the sample code I've been made available back in 2011. It doesn't work with ColdFusion 9 or 10. I've also listed all of the official "built-in" (but not supported) date formats.
http://pastebin.com/aQnembR3
<cfset q = queryNew("Name,Date", "varchar,date")>
<cfloop index="x" from="1" to="10">
<cfset queryAddRow(q)>
<cfset querySetCell(q, "Name", "Name #x#")>
<cfset querySetCell(q, "Date", now())>
</cfloop>
<cfset dfStyle = {dataformat = "m/d/yy h:mm"}>
<cfset o = spreadsheetNew('dateTest', true)>
<cfset spreadsheetAddRows(o, q)>
<cfset spreadsheetFormatColumn(o, dfStyle, 2)>
<cfset bin = spreadsheetReadBinary(o)>
<cfheader name="Content-Disposition" value="attachment; filename=dateTest_#dateFormat(now(), 'ddmmmyyyy')#.xlsx">
<cfcontent type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" variable="#bin#" reset="true">
I am using a cfloop to generate the titles of a bunch of variables.
Problem is on output I just get the actual variable name ie #qQuery.varName# instead of its value ie "Item Name".
Heres a quick taste of my code:
<cfloop query="qQuery">
<cfloop query="qTest">
<cfset varTest = "qQuery." & varName>
<cfoutput>#varTest#</cfoutput>
</cfloop>
</cfloop>
Thanks :)
Try something like this:
<cfset vartest = qQuery[varName]>