In Cognos, I would like to write 100 tables output to 100 flat files. I am planning to create a Project, which will be scheduled everyday. When it runs, it will write tables output to flat files.
Can you please tell me whether this is possible or not. If so, can you please specify the approach to solve this problem?
Thanks
Ram
Create a report for each table. I presume you know how to do it.
On a Cognos Connection portal create a job with all your reports and set up out put for all your reports.
Schedule execution of your new job.
1 task described in Report Studio User guide (And Framework Manager Developer Guide if you don't have model for your table)
2-3 in Cognos connection User Guide. Check out "Use Jobs to Schedule Multiple Entries"
Related
I use Power Automate to refresh a PowerBI dashboard when I update its input files on Sharepoint.
Since the inputs are generated via another automated process, the updates follow each other very closely, which triggers the start of one Power Automate job per updated file.
This situation leads to failed jobs since they are all running at the same time. Even worse, the first job succeeds but the refresh then happens when not all files are updated yet. It is also a waste, since I would only need one to run.
To accomodate for that, I tried to introduce a delay in the job. This makes sure that when the first job runs, refreshing powerBI will work. Howerver, the subsequent runs all fail, so I would still like to find a way not to run them at all.
Can anyone guide me in the right direction ?
For this requirement, you can set the Share Point trigger can run only 1 instance at same time. Please refer to the steps below:
1. Click "..." button of the trigger and click "Settings".
2. Enable "Concurrency Control" limit and set Degree of Parallelism as 1.
Then your logic app can not run multiple instances at same time.
Just checked my Kentico database (Azure hosting) and it ballooned to 21GB. This happened fairly recently since 4 months ago it was just a bit above 1GB.
Checked the tables and my Event Log table has over 2,000,000 entries!!!
Nothing has changed recently, my settings under Settings -> System -> Event Log are still the same:
Event Log Size: 1000
Since globals are also set to 1000, usually I have 2000 or so entries in the event log table.
Anyone knows what happened here? And how to stop it from happening?
If you have online marketing turned on and have a popular site, there will be lots of data in the OM_ tables. But still 20GB sounds huge, are there lots of asset files been added to the Content Tree, like videos? Also, is the database set to log all transactions? What kind of log files are popular? Errors or Information logs? Do you have some custom code that could produce lots of logs?
You can also email Kentico support to get a "check big tables" SQL script which can help you find out which are the large tables.
You should look into a few other areas as well. Having 2MM event log records won't cause a 20GB jump in DB size from a Kentico perspective as the event logs are pretty minimal data.
Take a look in the analytics, version history, email queue, web farm and scheduled task tables. Also check out the recycle bin. Are you integrating with any other system or inserting/updating a lot of data via the API? If so, this could cause a lot of transactional log files to build up. With Azure SQL I don't know of a way to clean those up.
My suggestion is to check other tables and not just the event log. Maybe query the event log manually via SSMS and see what the top 100 events are and that might help you find the problem. If you need to you could probably clear the log too either through the UI or manually truncate the table using SSMS.
I run 100sof computer in a network and 100sof user access those machines. Every day, thousands or more syslogsare generated from all those machines. Syslog could be any log including system failures, network, firewall, application errors etc.
Sample log would look like below
May 11 11:32:40 scrooge SG_child[1829]: [ID 748625 user.info] m:WR-SG-BLOCK-111-
00 c:Y th:BLOCK , no allow rule matched for request with entryurl:http:url on
mapping:bali [ rid:T6zcuH8AAAEAAGxyAqYAAAAQ sid:a6bbd3447766384f3bccc3ca31dbd50n ip:192.24.61.1]
From the logs, I extract fields like Timestamp, loghost, msg, process, facility etc and store them in HDFS. Logsare stored in json format. Now, I want to build a system where I can type a query in a web application and do analysis on those logs. I would like to be able to do queries like
get logs where the message contains "Firewall blocked" keywords.
get logs generated for the User Jason
get logs containing "Access denied" msg.
get log count grouped by user, process, loghost etc.
There could be thousands of different types of analytics I want to do. To add more, I want the combined results of historical data and the real time data i.e. combining batch and realtime results.
Now my questions is
To get the batch result, I need to run the batch spark jobs. Should I
be creating batch jobs for every unique query user makes. If I
do so, I will end up creating 1000s of batch jobs. If not, what kind
of batch jobs should I run so that I can get results for any type of
analytics.
Am I thinking it the right way. If my approach itself is wrong, then do share what should be the correct procedure.
While it's possible (via thrift server for example), Apache Spark main objective is not to be a query engine but building data pipelines for stream and batch data sources.
If the transformation you are only projecting fields and you want to enable ad-hoc queries, sounds like you need another data store - such as ElasticSearch for example. The additional benefit is that it comes with a Kibana that enable analytics to some extent.
Another option is to use a SQL engine such as Apache Drill.
Spark is probably not the right tool to use unless the size of these logs justifies the choice.
Are these logs in the order of a few gigabytes? Then use splunk.
Are these logs in the order of hundreds of gigabytes? Then use elasticsearch with maybe Kibana on top.
Are they in the order of terabytes? Then you should think to some more powerful analytical architecture and there are many alternatives here that basically do batch jobs in the same way as you would with Spark, but usually in a smarter way.
In Cognos environment there are many reports scheduled to be delivered to user's email id. How can I track all those reports without doing lot's of manual work.
Using audit package's Run Jobs and Run Reports I am not able to track reports which are scheduled to email burst as Run Report doesn't provide information on mode of bursting of a report.
Thanks in advance.
The recipients are not directly stored in the content store anywhere... they are pulled from a query within the report.
If the report is saving output, you can set it to retain the saved outputs for a few days. Every version that was sent out will be saved.
Outside that, you will have to do manual work. Get into the report, find the query that feeds into the burst, and run the query manually to see who it is built for.
If you have multiple burst reports in your environment, consider switching to use a table that stores all of your burst recipients. Then performing monitoring/maintenance of burst users will be a simple query away.
We are using SQL Azure for our application and need some inputs on how to handle queries that scan a lot data for reporting. Our application is both read/write intensive and so we don't want the report queries to block the rest of the operations.
To avoid connection pooling issues caused by long running queries we put the code that queries the DB for reporting onto a worker role. This still does not avoid the database getting hit with a bunch of read only queries.
Is there something we are missing here - Could we setup a read only replica which all the reporting calls hit?
Any suggestions would be greatly appreciated.
Have a look at SQL Azure Data Sync. It will allow you to incrementally update your reporting database.
here are a couple of links to get you started
http://msdn.microsoft.com/en-us/library/hh667301.aspx
http://social.technet.microsoft.com/wiki/contents/articles/1821.sql-data-sync-overview.aspx
I think it is still in CTP though.
How about this:
Create a separate connection string for reporting, for example use a different Application Name
For your reporting queries use SET TRANSACTION ISOLATION LEVEL SNAPSHOT
This should prevent your long running queries blocking your operational queries. This will also allow your reports to get a consistent read.
Since you're talking about reporting I'm assuming you don't need real time data. In that case, you can consider creating a copy of your production database at a regular interval (every 12 hours for example).
In SQL Azure it's very easy to create a copy:
-- Execute on the master database.
-- Start copying.
CREATE DATABASE Database1B AS COPY OF Database1A;
Your reporting would happen on Database1B without impacting the actual production database (Database1A).
You are saying you have a lot of read-only queries...any possibility of caching them? (perfect since it is read-only)
What reporting tool are you using? You can output cache the queries as well if needed.