Updating multiple worksheets in google spreadsheet - python-3.x

I have some code that look this:
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
SCOPES = ['https://www.googleapis.com/auth/spreadsheets']
service = build('sheets', 'v4', credentials=creds)
spreadsheet = {
'properties': {
'title': 'Data Integrity Report Completed on {}'.format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
}
}
spreadsheet = service.spreadsheets().create(body=spreadsheet,
fields='spreadsheetId').execute()
gsheet_id = spreadsheet.get('spreadsheetId')
response_date = service.spreadsheets().values().append(
spreadsheetId=gsheet_id,
valueInputOption='RAW',
range='A1:Z100',
body=dict(
majorDimension='ROWS',
values=miss_occ_df.T.reset_index().T.values.tolist())
).execute()
This code basically creates a google spreadsheet and appends my dataframe to the first worksheet. What I want is to have a spreadsheet that has 3 worksheets. I also need to name each worksheet and upload 3 different dataframes to each worksheet. How can I do this?

You want to achieve the following things.
Create new Spreadsheet.
You want 3 sheets in the created Spreadsheet.
You want to rename the sheets.
Put the values to each sheet.
You want to achieve them using google-api-python-client with Python.
If my understanding is correct, how about this modification? I think that your goal can be achieved by one API call. But in your case, it seems that 2 dimensional array for the request body is required to be used. So in this answer, I would like to propose the method for achieving your goal by 2 API calls. So please think of this as just one of several answers.
The flow of this method is as follows.
Flow:
Create new Spreadsheet.
At that time, the 3 sheets (worksheets) are created by giving the names. In this case, the method of create() is used.
Put the values to 3 sheets using the method of values().batchUpdate().
In your case, the values are put to the new Spreadsheet. So the values can be put using this method.
Modified script:
Please modify the script below service = build('sheets', 'v4', credentials=creds) as follows. And please set variables of sheet names and values.
# Please set worksheet names.
sheetNameForWorksheet1 = "sample1"
sheetNameForWorksheet2 = "sample2"
sheetNameForWorksheet3 = "sample3"
# Please set values for each worksheet. Values are 2 dimensional array.
valuesForWorksheet1 = miss_occ_df.T.reset_index().T.values.tolist()
valuesForWorksheet2 = miss_occ_df.T.reset_index().T.values.tolist()
valuesForWorksheet3 = miss_occ_df.T.reset_index().T.values.tolist()
spreadsheet = {
'properties': {
'title': 'Data Integrity Report Completed on {}'.format(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
},
"sheets": [
{
"properties": {
"title": sheetNameForWorksheet1
}
},
{
"properties": {
"title": sheetNameForWorksheet2
}
},
{
"properties": {
"title": sheetNameForWorksheet3
}
}
]
}
spreadsheet = service.spreadsheets().create(body=spreadsheet, fields='spreadsheetId').execute()
gsheet_id = spreadsheet.get('spreadsheetId')
batch_update_values_request_body = {
"data": [
{
"values": valuesForWorksheet1,
"range": sheetNameForWorksheet1,
},
{
"values": valuesForWorksheet2,
"range": sheetNameForWorksheet2,
},
{
"values": valuesForWorksheet3,
"range": sheetNameForWorksheet3,
}
],
"valueInputOption": "USER_ENTERED"
}
response = service.spreadsheets().values().batchUpdate(spreadsheetId=gsheet_id, body=batch_update_values_request_body).execute()
Note:
This modified script supposes that you have already been able to put and get values to the Spreadsheet using Sheets API.
References:
Method: spreadsheets.values.batchUpdate
Method: spreadsheets.create
At first, please confirm whether my understanding for your question is correct. If I misunderstood your question and this was not the direction you want, I apologize.

Related

How to copy a Google sheet and get data validation to stay intact?

I have a Google spreadsheet template with two sheets, Data Entry and Data Validation. The Data Validation sheet has a number of columns with valid values for matching columns on the Data Entry sheet. Everything works as expected. I need to copy these two sheets to a Sheet Under Test (SUT). I am using the sheets API to copy both sheets. I copy the Data Validation sheet first and then the Data Entry sheet. Here's the code and this appears to work.
const request = {
spreadsheetId :fromSpreadsheetId,
sheetId : fromSheetId,
resource:{
destinationSpreadsheetId: toSpreadsheetId,
},
}
const result = await _sheetService.spreadsheets.sheets.copyTo(request)
On my SUT, both sheets appear and the Data entry sheet has all the expected dropdowns and they all have the proper values. Seems perfect. The problem is when you select an item from any drop down in any column it selects and enters the proper value and then adds a red triangle and the message that an invalid value has been entered. If the column has the rejection setting then the value is removed and the error dialog appears.
The image shows two cells where I have already selected Video Course from the drop down.
If I go in and reselect column where I want validation, use Data→DataValidation… and just hit the Save button that column starts working, so it would appear everything came across correctly but the sheet doesn't think so. Is there any programmatic way to force the above process that I did manually? Is there something else I need to do in the sheets.copyTo method to make this work properly?
EDIT
This project is written in Node.js with a combination of TypeScript and JavaScript. The lower level code for talking to the Sheets API and copying the sheet between spreadsheets can be found in this github file. The method is copySheetFromTo and it's at the bottom of the file.
A sample source sheet with public view permissions
A sample destination sheet with public edit permissions
The integration test that used the above two files to copy the sheets is at the bottom of the file and has 'DEBUGGING TEST' at the beginning of the name (starts at line 209)
You want to copy the sheet including Data Validation.
When the copied sheet is used, an error occurs at the drop down menu of Data Validation.
You want to remove this error.
If my understanding is correct, how about this answer? In this answer, in order to remove the error, I overwrite the Data Validation of the copied sheet as a workaround. Please think of this as just one of several workarounds.
The flow for your situation is as follows.
Flow:
Retrieve all data validations from the sheet of Data Entry in the source Spreadsheet ("DataValidationTest") using the spreadsheet.get method.
Copy the sheets of Data Entry in the source Spreadsheet ("DataValidationTest") to the destination Spreadsheet ("Public Destination Sheet").
After the sheets of Data Entry was copied, rename the sheet name from Copy of Data Entry to Data Entry.
Then, overwrite the retrieved data validations to the sheet of Data Entry using the spreadsheet.batchUpdate method.
In this case, the structure of data validations retrieved by the spreadsheet.get method is almost the same with the structure for the spreadsheet.batchUpdate method. This workaround used this.
Copy the sheets of Data Validation in the source Spreadsheet ("DataValidationTest") to the destination Spreadsheet ("Public Destination Sheet").
Rename the sheet name of Copy of Data Validation to Data Validation.
Sample script:
When you test this script, please set the variables. And I think that sheet of sheet.spreadsheets.get(), sheet.spreadsheets.batchUpdate() and sheet.spreadsheets.sheets.copyTo() is the same with sheetOps of your script.
const srcSpreadsheet = "###"; // Please set this.
const tempDestSheetId = "###"; // Please set this.
const srcDataEntrySheetId = 0; // Please set this.
const srcDataValidationSheetId = 123456789; // Please set this.
let dataValidation = await sheet.spreadsheets.get({
spreadsheetId: srcSpreadsheet,
ranges: ["Data Entry"],
fields: "sheets/data/rowData/values/dataValidation"
});
let data = dataValidation.data.sheets[0].data;
let rows = [];
for (let i = 0; i < data.length; i++) {
if (data[i].rowData) {
rows = data[i].rowData;
break;
}
}
sheet.spreadsheets.sheets.copyTo(
{
spreadsheetId: srcSpreadsheet,
sheetId: srcDataEntrySheetId,
resource: { destinationSpreadsheetId: tempDestSheetId }
},
(err, res) => {
sheet.spreadsheets.batchUpdate(
{
spreadsheetId: tempDestSheetId,
resource: {
requests: [
{
updateSheetProperties: {
fields: "title,sheetId",
properties: { sheetId: res.data.sheetId, title: "Data Entry" }
}
},
{
updateCells: {
rows: rows,
range: { sheetId: res.data.sheetId },
fields: "dataValidation"
}
}
]
}
},
(er, re) => {
if (err) {
console.error(er);
return;
}
console.log(re.data);
}
);
}
);
let result1 = await sheet.spreadsheets.sheets.copyTo({
spreadsheetId: srcSpreadsheet,
sheetId: srcDataValidationSheetId,
resource: { destinationSpreadsheetId: tempDestSheetId }
});
let result2 = await sheet.spreadsheets.batchUpdate({
spreadsheetId: tempDestSheetId,
resource: {
requests: [
{
updateSheetProperties: {
fields: "title,sheetId",
properties: {
sheetId: result1.data.sheetId,
title: "Data Validation"
}
}
}
]
}
});
console.log(result2.data);
Note:
This script supposes that Sheets API has already been able to be used.
This is the simple modified script for showing the flow of workaround. So please modify this for your situation.
As an important point, in order to overwrite the data validations to the copied sheet of Data Entry, it is required to execute it after the copy of the sheet of Data Entry was completely finished. So I modified the script like above.
References:
Method: spreadsheets.batchUpdate
Method: spreadsheets.get
Method: spreadsheets.sheets.copyTo

What is the best way to load JSON from a dynamic filename in node.js & Typescript?

I am creating a simple web application with node.js and Typescript to build my familiarity with it, and I want to know the best way to load a JSON config file whose filename is determined at runtime.
I have found a few suggestions online:
Using the fs module in node
Importing the JSON file like a module
The first approach appears like it should work, but the second seems a lot neater. The problem is that with the second approach, I have the following error:
An import declaration can only be used in a namespace or module. ts(1232)
because I want to import the file in a constructor with a filename specified as an argument. Secondly I get the following error if I try to append the given filename to the constant directory I want to get it from:
';' expected. ts(1005)
Here is a (non-compiling) code snippet from the class in which I'm trying to load the JSON, and an example JSON file that I'm trying to load in.
Typescript class:
import Floor from './Floor'
import Elevator from './Elevator'
import Person from './Person'
class Building {
public name: string
private floors: Floor[]
private elevators: Elevator[]
private people: Person[]
private time: number
constructor(config_filename: string) {
import * as config from '../config/'+config_filename
const building = (<any>config).building
this.name = name
this.floors = []
building.floors.forEach((floor) => {
this.floors.push(new Floor(floor.number, floor.name))
})
this.elevators = []
building.elevators.forEach((elevator) => {
this.elevators.push(new Elevator(elevator.name, elevator.weight_capacity, elevator.start_floor_no, this))
})
this.people = []
building.people.forEach((person) => {
const person_instance = new Person(person.name, 10, person.algorithm)
this.people.push(person_instance)
this.floors[person.start_floor_no].addOccupant(person_instance)
})
this.time = 0
}
...
Example JSON config file:
{
"building": {
"name": "Basic Inc.",
"floors": [
{
"number": 0,
"name": "Ground Floor"
},
{
"number": 1,
"name": "1st Floor"
}
],
"elevators": [
{
"name": "Bruce",
"weight_capacity": 100,
"start_floor_no": 0
}
],
"people": [
{
"name": "Wendy Fox",
"start_floor_no": 0,
"algorithm": "desk"
}
]
}
}
Should I stick with the first approach, or is there some neater way to load the JSON file with a filename only known at runtime?
In your example, the import statement applies to when the TS is 'compiled' into JS, not at runtime with new Building(<name>) (when the code is actually executed, after compilation by the Node process).
Option 1 is a good way to do it for Node. I imagine you're trying to get away from the awkward fs functions.
Another option is to use a GET request from the code to itself, although it is ostensibly the same, you can easily explore the async / await functions which are a bit neater*.
*(last time a few months ago I used fs I did try but failed to get it going with async/await, but possibly this has changed now?)

CDON API RESTful Api GET request

I'm currently working on fetching customer data from cdon, it's an e-commerce platform. They have their API documentation here:
CDON Api Docu
First let me show you my code:
myToken = '<token here>'
myUrl = 'https://admin.marketplace.cdon.com/api/reports/d8578ef8-723d-46cb-bb08-af8c9b5cca4c'
head = {'Authorization': 'token {}'.format(myToken),
'Status':'Online',
'format':'json'}
filters = '?filter={"Status":["Online"],"format": ["json"] }}'
response = requests.get(myUrl + filters, headers=head)
report = response.json()
print(report.products)
This is returning only the parameters. like for example at at this JSON: CDON Github
Status has a value Online this online is a group of itemsthat I only want to get.
What I'm trying to get is a response like this:
{
"Products": [
{
"SKU": "322352",
"Title": "Fabric Cover",
"GTIN": "532523626",
"ManufacturerArticleNumber": "",
"StatusCDON": "Online",
"ExposeStatusCDON": "Buyable",
"InStock": 0,
"InStockCDON": 0,
"CurrentPriceSE": null,
"OrdinaryPriceSE": null,
"CurrentPriceCDONSE": 299.0000,
"OrdinaryPriceCDONSE": null,
"CurrentPriceDK": null,
"OrdinaryPriceDK": null,
"CurrentPriceCDONDK": null,
"OrdinaryPriceCDONDK": null,
"CurrentPriceNO": null,
"OrdinaryPriceNO": null,
"CurrentPriceCDONNO": null,
"OrdinaryPriceCDONNO": null,
"CurrentPriceFI": null,
"OrdinaryPriceFI": null,
"CurrentPriceCDONFI": null,
"OrdinaryPriceCDONFI": null
},
Which means the full list of the items that are Online
How should I put this... among all the API's I tried this one is very confusing, is this even RestFul? If I can achieve the python equivalent of this C# sample code:
public string Post(Guid repordId, string path)
{
var filter = new JavaScriptSerializer().Serialize(new
{
States = new[] { "0" } // Pending state
});
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair("ReportId", repordId.ToString()),
new KeyValuePair("format", "json"),
new KeyValuePair("filter", filter)
});
var httpClient = new HttpClient() { BaseAddress = new Uri("https://admin.marketplace.cdon.com/") };
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("api", ApiKey);
var response = httpClient.PostAsync(path, content).Result;
response.EnsureSuccessStatusCode();
return response.Content.ReadAsStringAsync().Result;
}
I may be able to undestand how this API works, the response that I got was taken manually from their report function in JSON format.
Image
I made many attempts and at that code ( my code ) I stopped, being on this for 4 hours made me give up and ask. Trust that I have searched as many references as I could. It's really confusing.
How do I get the response that I want? Filtering via url? or via header? is this even restful? Help T_T
The documentation states in the first line, emphasis mine:
In order to generate a report you perform a POST call to the reports API with the parameters you wish to use for the report.
Your Python code does not make a POST request, you are trying a GET request. The documentation goes on
[...] to filter on Swedish orders you set the CountryCodes
attribute to “Sweden” and to get returned and cancelled orders you set
the States attribute to 2 and 3. So in the end the filter would look
like this:
{
"CountryCodes": [ "Sweden" ],
"States": ["2", "3"]
}
So you need to prepare a filter object (a dictionary in Python) with the filters you want. Luckily the Python syntax for dictionaries is equivalent (Python is flexible and also allows single-quoted strings):
filter = {
'CountryCodes': [ 'Sweden' ],
'States': [ '0' ]
}
The documentation goes on
You then post the parameters as form data (content-type:
application/x-www-form-urlencoded) so the request body would look like
this:
ReportId=d4ea173d-bfbc-48f5-b121-60f1a5d35a34&format=json&filter={"CountryCodes":["Sweden"],"States":["2","3"]}
application/x-www-form-urlencoded is the default for HTTP post, the requests module knows that and does this for you automatically. All you need to do is to prepare a data dict which will contain the data you want to post.
data = {
'ReportId': 'd4ea173d-bfbc-48f5-b121-60f1a5d35a34',
'format': 'json'
'filter': json.dumps(filter)
}
The filter parameter is supposed to be in JSON format. You must encode that yourself via json.dumps().
import json
head = { ... as above }
filter = { ... as above }
data = { ... as above }
response = requests.post(url, data, header=head)
I'll leave figuring out setting the Authorization header properly as an exercise for you. Partly because it isn't hard, partly because I have no intention of creating an API key with this website just for testing this and partly because it's entirely possible that your current header already works.

load json into excel using office js web addin

if I have json string which looks like this: [{"id":1,"name":"manish"},{"id":1,"name":"John"}]
can I just office js to simply load it in table. I saw this https://learn.microsoft.com/en-us/office/dev/add-ins/excel/excel-add-ins-tables#import-json-data-into-a-table
but then when I add more columns in my json I will have to do code changes and code is not generic enough. I could live with it but was wondering if there is a better way.
You can use Object.keys(obj).length to count the number of properties in one of your JSON objects. Something like myObjects[0].keys(obj).length.
Then get a Range object for the cell that will be the upper-left cell of the table; for example, getRange("A1").
Then use that Range object's getResizedRange method and pass 1 as the first parameter (rows) and pass the number of properties in your JSON object as the second parameter (columns).
Use the Range object that is returned by getResizedRange as the first parameter to the sheet.tables.add method.
Expanding on #RickKirkham answer and the link in the OP from https://learn.microsoft.com/en-us/office/dev/add-ins/excel/excel-add-ins-tables#import-json-data-into-a-table I build a "more generic" approach. My issue was that I have no control over the JSON response, I don't know how many rows, columns etc. I first tried to just use A1 and expected it to expand as needed, but that didn't work.
I did finally get it to work as I wanted below:
var transactions = [
{
"DATE": "2017",
"MERCHANT": "The Phone Company",
"CATEGORY": "Communications",
"AMOUNT": "120"
},
{
"DATE": "2017",
"MERCHANT": "Southridge Video",
"CATEGORY": "Entertainment",
"AMOUNT": "40"
}
];
var sheet = context.workbook.worksheets.getItem("Sheet1");
var rng = sheet.getRangeByIndexes(0, 0, 1, Object.keys(transactions[0]).length)
var expensesTable = sheet.tables.add(rng, true);
expensesTable.name = "ExpensesTable";
expensesTable.getHeaderRowRange().values = [Object.keys(transactions[0])];
for (let i = 0; i < transactions.length; i++) {
expensesTable.rows.add(null, [Object.values(transactions[i])]);
}
if (Office.context.requirements.isSetSupported("ExcelApi", "1.2")) {
sheet.getUsedRange().format.autofitColumns();
sheet.getUsedRange().format.autofitRows();
}
sheet.activate();
PS: Very new to JS/Office JS atm.

Need to get the values of the objects (JSON) which have been created dynamically

How to get the top object value in PentahoDI? I have got the other elements like Category, Subcategory, section from the following example of Json file. However, I need to capture the first root object which is x#chapter#e50de0196d77495d9b50fc05567b4a4b and x#e50de0196d77495d9b50fc05567b4a4b
{
"x#chapter#e50de0196d77495d9b50fc05567b4a4b": {
"Category": "chapter",
"SubCategory": [
"x#4eb9072cf36f4d6fa1e98717e6bb54f7",
"x#d85849fbde324690b6067f3b18c4258d",
"x#3edff1a1864f41fe8b212df2bc96bf13"
],
"Section": {
"display_name": "Week 1 Section"
}
},
"x#e50de0196d77495d9b50fc05567b4a4b": {
"category": "course",
"Subcategory": [
"x#e50de0196d77495d9b50fc05567b4a4b"
],
"Section": {
"advanced_modules": [
"google-document"
],
}
}
}
In the Fields tab of the Json Input step I have given the Names and Paths as: Category --> $..Category, Subcategory --> $..Subcategory, Section --> $..Section.
However, I am unable to get the root element as it is crucial information for us to work on it. ex (x#chapter#e50de0196d77495d9b50fc05567b4a4b and x#e50de0196d77495d9b50fc05567b4a4b)
I have used the following code to get the values of the dynamic objects but it didnt work. The following is the code I used it.
var obj = JSON.parse (JBlock) //Jblock is the one which holds the entire string.
var keys = Object.name( obj);
JSONPath is not able to get the keys of a JSON structure. This is one of my main issues with JSONPath, and I wish Pentaho had included other JSON parsing engines.
This JavaScript to be used in Modified Java Script Value works for me. Add a value in the fields editor like this:
And then a script like this:
var obj = JSON.parse(JBlock);
var keys = Object.keys(obj);
for (var i = 0; i < keys.length; i++) {
    var row = createRowCopy(getOutputRowMeta().size());
    var idx = getInputRowMeta().size();
row[idx++] = keys[i];
putRow(row);
}
trans_Status = SKIP_TRANSFORMATION;

Resources