Pipelines and PowerBI

I have not tried this but it appears you would use the realtime streaming dataset in PowerBI then use the endpoint created there in the webhook section on the MSV side.  

Check here for more details Power BI Real Time Streaming  

Hello @BrianK 

thank you for the link, now it's clear how to create a real time streaming.

However, during the creation process it is necessary to specify what data will be sent to Power BI through the REST API. 

For example:

I expect that MSV pipelines have already defined what fields will be sent to Power BI, so I think it is necessary to know these fields and their type to configure the integration.

Do you agree?

Would you be able to tell me a way to retrieve this information?

Thank you,
Paolo

Looking more into it appears the push semantic model is required on the PowerBI side,  which the details are available on that same link.  

Hi Paolo!

While I am uncertain of the livestream PowerBI capability, we do have a solution to allow configuring pipelines where you can select job results to send up to your BI Tool. This method, called Multi-site Reporting (MSR) uses a postgres connection from a designated secondary Director that will broker the traffic.

I've added some steps below as of how to configure this.

Prerequisites:

•  You must be able to utilize the MSR Feature, which may require a license. Please contact your CSM if you do not currently have this capability. Please note that the license is of no additional cost! It is only currently required as the feature is in limited availability as of now.
• MSR requires two Directors.
• It is recommended to have CA signed certificates when configuring this in a production environment.

Steps:

1. Deploy and license both directors. Your Reporting director will have all of the actors registered, while your central director simply acts as a pivot point to connect to postgres.
2. On the Central Director within the CLI, change directory to /opt/apps/verodin/planner/scripts/setup_msr and run sudo ./setup.rb
3. When prompted, enter the IP address of your PowerBI instance. Record the credentials. You will use these in PowerBI under Options and Settings > Data Source Settings to create your connection to the Central Director IP Address using port 5432.
4. Set up the Central Director Pipeline by logging into the UI, navigated to Settings > User Settings > User Groups and ensuring the proper pipeline permissions are checked.
5. Naviate to User > User Preferences, click 'Show Key' and copy the API Key.
6. In the Reporting Director UI, navigate to Jobs > Pipelines. Click Add Pipeline > Director. Enter the IP or FQDN, and paste in the API key copied in Step 5.
7. Lastly, after running a job, open the job results page, click the kabob menu and hit "Publish Job Result". The results will then populate within the database created in PowerBI.

I hope this helps!

Hello @BrianK and @lnhymon and thanks for your answer.

@lnhymon we only have one on-prem Director, so I can't apply your solution. Furthemore I didn't understand if the PowerBI instance must be local or it can be in the cloud.

@BrianK I understood you suggested the use of push semantic model. The question is exactly about that semantic model. When you try to configure it you get from Power BI the request to explicitly list all the fields you will receive from the stream, i.e. from MSV. This is why I asked for an example of json that MSV produce when you click on the "Publish" button on a job result page.

Thank you.

Paolo

Hi Paolo,

If there is appetite to go the route of MSR and getting a second director stood up, please consult your CSM.

Else, to answer your question, here is a screenshot of the schema I retrieve from MSV in PowerBI when I publish data to it: 

If there is a better way I can export this to make it more usable, please let me know and I'll see what I can do. Otherwise, let me know if this is what you were looking for or if there should be some other data I can grab for you.

Hello @lnhymon , thank you for your response.

I will try to investigate with our CSM what it means to have two Directors and the MSR license.

As for your screenshot, I don't know if it can be useful to me, because from what I understand I should configure in the PowerBI streaming dataset every single field that MSV will send to the API. What I see in your screenshot instead seems to me tables that, I guess, inside will have a whole series of fields.

However, before proceeding, I would like to share with you this simple scheme, to confirm if I have understood well the operation of the pipelines and their integration with PowerBI.

What I have deduced is that MSV with pipelines can invoke a PowerBI API to "pass" job data. It seems to me that it can pass them in JSON format, but I have not been able to confirm this hypothesis.

Can you confirm that pipelines work in this way? Or is the mode with two Directors and the MSR license the only possible one?

Second question: in order to publish data on an external service like PowerBI, our Director would need to go through a proxy. Looking at the Proxy Rules in Director Settings, I have not found a way to configure the proxy for pipelines (as it is possible to do for integrations or Operational Status webhook). Is there a way to do this or is it not currently available?

Last question: is there a dedicated log files on the Director where to find all the operations related to publishing data to pipelines?

Thank you very much

Paolo

Second question: in order to publish data on an external service like PowerBI, our Director would need to go through a proxy. Looking at the Proxy Rules in Director Settings, I have not found a way to configure the proxy for pipelines (as it is possible to do for integrations or Operational Status webhook). Is there a way to do this or is it not currently available?

@lnhymon wrote:

For the second question, unfortunately, there is not currently a way to do this, but we have received this request from other customers and entered it as an RFE (Request For Enhancement). I can update this thread once that RFE has been complete and the feature is available.

---

Hi @lnhymon is there some news about this RFE?

Thank you

Paolo

@paolocarrara so sorry for the delay - still no update on this at this moment. Will continue to keep an eye on the status of the request. Thank you so much for your patience. 

Hi @lnhymon !

We workarounded the problem of the proxy for the Director, so we do not need anymore a proxy to use pipelines. Moreover, we are adopting Chronicle, so my new question is: do you have some link/docs where the use of pipelines with Chronicle is explained? Or, can you describe what is needed to use Pipelines with Chronicle?
Thank you,

Paolo

Hi @paolocarrara ! It's great to hear from you again!

For my curiosity, was this workaround a technical workaround done in MSV, or an organizational exception? Either way, glad to know it's working.

As for your question for pipelines in chronicle, nothing comes to mind, but I can do some research and see if I can find anything for you. I appreciate your patience!

Hi @lnhymon,

the "workaround" is at network level. We changed the Director's default route to a firewall that let the Director connect to internet (only outbound with detailed ACLs). Then we configured static routes for internal resources and actors. In this way we can "bypass" the use of a proxy to go outside.

Please let me know if you have some use cases to use pipelines with Chroncle because I think it can be very useful for our activities and can give more values to MSV and Chronicle too.

Thank you,

Paolo