Since I have little experience programming, I first tried posting this "job" on a freelance website. Then, 4 programmers who seemed to what they were doing failed to solve it (maybe they didn't know what they were doing). After this, I decided to attempt it myself, and that is why I came to Stack Overflow, which I believe will point me in the right direction.
The problem appears quite simple: the program in question gives me rows and columns of data, just like a spreadsheet. As time goes by, new rows are added on top. It looks like this:
I just need to replicate this data inside an Excel spreadsheet, so that I can perform analysis.
I will keep it short, as I don't know what further detail I could give. Perhaps looking at the program files could help in establishing what sort of program it is. Download link: http://xpproupdate.xpi.com.br/xppro.zip
Thanks!
Some loose ideas:
Method 1 (assuming this is an app connected to the Internet):
Try packet-sniffing. Instead of extracting the data from the app download a packet sniffing app and look and the data-flow. See on what port the app is exchanging data. If the data is not encryped the tasks should be fairly easy.
As a reference see this packet sniffer in C#:
http://www.codeproject.com/Articles/17031/A-Network-Sniffer-in-C
Method 2 (assuming no connection to the Internet, or if there is encryption involved):
If the data is encrypted or the app simply does not interact with the Internet then try to access the app's Win32 window handle and traverse it's internal controls.
Method 3 (last resort):
Frequent window image screenshot and scraping the data from the image using a simple OCR.
Related
The question here is: Am I on the right path (this is the first time I'm trying this), and if not, what would be smarter to try? If this is the right path, can you offer suggestions on how to do this best, because if this works, I am going to use it often on a lot of different tasks in this app.
I'm running a PowerApps Canvas app. As part of its program, I want it to be able to reference (read-only) a collection of data. That data is in ServiceNow, and my group is not permitted to access ServiceNow using the API.
During testing of the app, I just had it reference a SharePoint list (which I had filled with some dummy data), but I can re-code those lines as needed to pull from some other data source.
Because I am touching a few different systems here, I am not sure if this is the right way to go and I'm afraid I'll spend too long trying and find out that it would never have worked because of x. Thus my question.
This is what I think will work. Am I headed in the right direction?
Set up the scheduled report in ServiceNow. (Done!)
Program ServiceNow to email the Excel file output. Make sure it is
always the same title. (Done!)
Build a Power Automate flow to capture that email and save the
attached file to a location (OneDrive?) that can be accessed by the
app. If there is a file there already, delete it first.
Add the Excel file as a data source to the app, and start
referencing it as needed.
8-12 hours later, ServiceNow pushes out another scheduled data
drop, and the whole thing updates again.
In my perfect world, this system would work completely unattended.
Offhand, a glitch I can see is that ServiceNow generates an Excel file, but it's not a table, and PowerApps I think must ingest as a data source an Excel file that is a table. But (shrug) I might be wrong.
Am I thinking of this correctly? Is this the best avenue to follow?
I try to share my position to another secured device like my second phone. I use MapBox and AndroidStudio and I'm able to see my position but only from my phone. I try to find a solution that helps me in my problem. So I want to know if it's possible to do that.
Sincerely, Tony
I see that you are new here. I would recommend you not to ask questions about problems that you have not yet properly researched on your own.
It would help if you could share at least your thoughts on how you would approach this task. Also, make sure to answer specific questions. Otherwise you will get unspecific answers, like the one below.
Now to your problem:
This definitely is possible and there are many ways to accomplish this. The approaches to accomplish this depend on means to your disposal. Do you have a webserver that you can utilize? Or would you like to transmit the position directly from one device to another?
Via Webserver:
Create Webserver side script that listens to HTTP POST requests and writes the POST parameter (your position) to a database/file.
Create a second script that will answer a request for this position.
Call script 1 with Device 1, that wants to share its position.
Call script 2 with Device 2, that wants to read position.
Display position on map in application on Device 2.
Direct, one device to another:
You could even send your position via text message, and make your mobile application read the message, then display the position on a map.
I am working on a scraper that is growing bigger and bigger and I'm worried about making the wrong design choices.
I have never done more than short scripts in python and I'm at a loss knowing how to design a project with bigger proportions.
The scraper retrieves data from different, but similar themed websites, so an implementation for each site is needed.
The desired raw text of each website is then put through a parser which searches for the required values.
After retrieving the values they should be stored in a 3N-Database.
In its final evolution the scraper should run on a cloud service and check all the different sites periodically for new data. Speed and performance are not of highest importance but desirable. Most importantly the required data should be retrieved without unnecessary reuse of code.
I'm using the Selenium webdriver and have the driver object implemented as a singleton, so all the requests are done by the same driver object. The website text is then part of state of that object.
All the other functionality is currently modelled as functions, everything in one file. For adding another website to the project I first copied the script and just changed the retrieval part. As it soon occurred to me that that's pretty stupid I wanted to ask for design recommendations.
Would you rather implement a Retriever mother class and inherit from that for every website or is there an even better way to go?
Many thanks for any ideas!
So a bit of a general question. I work as a data analyst for a startup. My primary process involves taking existing customer data a client has and cleansing/normalizing it to fit into our platform once as part of our onboarding process. A member of our team exports their data from their system they are transitioning from or, if they kept track of it in house, we receive their Excel log they used to track it. It is always in a different format and requires extensive cleansing (avg 1 min/record). We take what is usually one large table (.xlxs format), and after cleansing, split it into four .csv files; which we load as four tables on our platform.
I feel I have optimized the process quite well in terms of the process steps and cleansing with excel functions (if, concat, text-to-columns, etc). I have beginner-intermediate skills in VBA and SQL and have just scratched the surface in R; what is frustrating is that I know there is the potential to automate this process but I just don't know where to start. If anyone has experience with something like this, code, a link to an article / another thread, or just some general direction would be much appreciated. Please ask for clarification where you feel it is needed. Thanks.
This will be really hard to do in Excel. If you have the time you can try out Optimus, a Data Cleansing library written in Python and Pyspark (you don't need to know spark). Here is the webpage https://hioptimus.com.
You can create Data Pipelines with it, and I recommend that you do that, try to generalize your processes, and asking the client for more a structure way of passing the data.
The good thing is that you don't need Big Data for running Optimus, bit if you have it some day, the same code will work.
Check out the documentation for more:
http://optimus-ironmussa.readthedocs.io/en/latest/
Let me know if you have doubts!
We are bumping into limitations with Flurry. We use events and parameters to track some game play info (like number of KO/map) but 1/ the limit of 15 parameters per event is a problem and 2/ the visualisation is not good (for instance Ko/map is shown by map so we have to open each event one after another).
We are trying to build a better visualisation with excel using the CSV files provided by Flurry, but then again we need to download the 50+ CSV files and it's really not convenient.
Is there a way to get all the information in one CSV or to get the information another way?
As a side note Flurry support is not answering any of our emails. :(
thanks for your help!
Have you tried checking out playtomic instead. Sounds like it might match your problem better.
They have an API to access your data. So you should be able to access it realtime.
You might also want to check out www.parse.com