Store historic data after each dataset refresh

Stefan2902 — Thu, 18 Jan 2018 13:23:12 GMT

Hi all,

I have been creating a KPI dashboard which is scheduled to run every day. The result of each KPI is something like:

a) how many data points have been loaded

b) what time has it taken to load it

In order to generate some statistics and averages I now want to save these values after each report execution / dataset refresh.

Having said that I am struggling to find a meaningful way how to export this "historic data" every day e.g. to CSV / Excel Workbooks etc. so that I can reuse is next reporting run.

I have found an example in the internet using R but since I do not have any experience around this I was wondering if there is an easier way how to achieve thist (e.g. M function to export to CSV)?

BR,

Stefan

Re: Store historic data after each dataset refresh

Stefan2902 — Thu, 18 Jan 2018 16:26:11 GMT

Dear all,

I understood that most likely using R is the only way how to achieve this. I have now played around and created the following R script which runs after I have pulled together my dataset:

# 'dataset' holds the input data for this script
require(gdata)
write.table(trim(dataset), file="//Client/C$/Users/S8937/temp/test012.csv", sep = ",", row.names = FALSE, col.names=FALSE, append = TRUE)
plot(dataset)

Unfortunately I do get the following stack trace. Any idea on how to fix this?

Error Message:
Object reference not set to an instance of an object.

Stack Trace:
Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> Microsoft.Mashup.Evaluator.Interface.ErrorException: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.PowerBI.Radio.RConnection.get_ServerVersion()
   at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.LoadServerVersion()
   at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.LoadCachedServerVersion()
   at Microsoft.Mashup.Engine1.Library.Common.DbEnvironment.CreateConnection()
   at Microsoft.Mashup.Engine1.Library.Common.DbValueBuilder.ConnectionReader..ctor(DbEnvironment environment, Func`2 readerWrapper)
   at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.Initialize()
   at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.EnsureInitialized()
   at Microsoft.Mashup.Engine1.Library.Common.NativeQueryTableValue.GetEnumerator()
   at Microsoft.Mashup.Engine1.Runtime.TableValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Runtime.TableValue.get_Item(Value key)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.FieldAccessInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction1.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction1.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.Instruction.Execute(MembersFrame1& frame, Instruction[] instructions)
   at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(Value frame)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
   at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame2& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame2& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
   at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame2& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue2.Invoke(Value arg0, Value arg1)
   at Microsoft.Mashup.Engine1.Runtime.Extensibility.ExtensionFunctionValue.Invoke(Value[] args)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(MembersFrame1& frame)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Runtime.Library.Linker.BindFunctionValue.TypedInvoke(RecordValue environment, Value section, TextValue name)
   at Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue3`4.Invoke(Value arg0, Value arg1, Value arg2)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue0.Invoke()
   at Microsoft.Mashup.Engine1.Language.ListInstruction.RuntimeListValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.ListInstruction.RuntimeListValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Value arg0)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.FunctionInvocationInstruction2.Execute(Value frame)
   at Microsoft.Mashup.Engine1.Language.Instruction.ExecuteCondition(Value frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.ExecuteCondition(Value frame)
   at Microsoft.Mashup.Engine1.Language.IfInstruction.Execute(Value frame)
   at Microsoft.Mashup.Engine1.Language.DebugInstruction.Execute(Value frame)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(Int32 index)
   at Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.get_Item(Int32 index)
   at Microsoft.Mashup.Engine1.Language.MembersFunctionValue0.Invoke()
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallPartition.BeginGetResult[T](Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallDocumentEvaluator.Evaluation`1.OnBufferComplete(Exception exception)
   at Microsoft.Mashup.Evaluator.Firewall.BeginBufferPartitions(Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallDocumentEvaluator.BeginGetResultInternal[T](DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.Interface.IDocumentEvaluatorExtensions.GetResult[T](IDocumentEvaluator`1 evaluator, DocumentEvaluationParameters parameters)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetResult[T](IMessageChannel channel, BeginGetResultMessage message, Action`1 action)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetPreviewValueSource(IMessageChannel channel, BeginGetPreviewValueSourceMessage message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.OnMessageWithUnknownChannel(IMessageChannel baseChannel, MessageWithUnknownChannel messageWithUnknownChannel)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.EvaluationHost.Run()
   at Microsoft.Mashup.Evaluator.SafeThread2.<>c__DisplayClass15.<CreateAction>b__14(Object o)
   at Microsoft.Mashup.Container.EvaluationContainerMain.SafeRun(String[] args)
   at Microsoft.Mashup.Container.EvaluationContainerMain.Main(String[] args)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.Interface.EvaluationResult2`1.get_Result()
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.<>c__DisplayClass2.<BeginGetResult>b__0(EvaluationResult2`1 result)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.<>c__DisplayClass2.<BeginGetResult>b__0(EvaluationResult2`1 result)
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.SimpleDocumentEvaluator.BeginGetResult(DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallPartition.BeginGetResult[T](Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallDocumentEvaluator.Evaluation`1.OnBufferComplete(Exception exception)
   at Microsoft.Mashup.Evaluator.Firewall.BeginBufferPartitions(Action`1 callback)
   at Microsoft.Mashup.Evaluator.FirewallDocumentEvaluator.BeginGetResultInternal[T](DocumentEvaluationParameters parameters, Action`1 callback)
   at Microsoft.Mashup.Evaluator.Interface.IDocumentEvaluatorExtensions.GetResult[T](IDocumentEvaluator`1 evaluator, DocumentEvaluationParameters parameters)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetResult[T](IMessageChannel channel, BeginGetResultMessage message, Action`1 action)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetPreviewValueSource(IMessageChannel channel, BeginGetPreviewValueSourceMessage message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.OnMessageWithUnknownChannel(IMessageChannel baseChannel, MessageWithUnknownChannel messageWithUnknownChannel)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.EvaluationHost.Run()
   at Microsoft.Mashup.Evaluator.SafeThread2.<>c__DisplayClass15.<CreateAction>b__14(Object o)
   at Microsoft.Mashup.Container.EvaluationContainerMain.SafeRun(String[] args)
   at Microsoft.Mashup.Container.EvaluationContainerMain.Main(String[] args)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.Interface.EvaluationResult2`1.get_Result()
   at Microsoft.Mashup.Evaluator.RemotePreviewValueSource.<>c__DisplayClass5.<RunStub>b__3()
   at Microsoft.Mashup.Evaluator.EvaluationHost.ReportExceptions(IHostTrace trace, IEngineHost engineHost, IMessageChannel channel, Action action)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.EvaluationHost.<>c__DisplayClass7.<TryReportException>b__6()
   at Microsoft.Mashup.Common.SafeExceptions.IgnoreSafeExceptions(IEngineHost host, IHostTrace trace, Action action)
   at Microsoft.Mashup.Evaluator.EvaluationHost.TryReportException(IHostTrace trace, IEngineHost engineHost, IMessageChannel channel, Exception exception)
   at Microsoft.Mashup.Evaluator.EvaluationHost.ReportExceptions(IHostTrace trace, IEngineHost engineHost, IMessageChannel channel, Action action)
   at Microsoft.Mashup.Evaluator.RemotePreviewValueSource.RunStub(IEngineHost engineHost, IMessageChannel channel, Func`1 getPreviewValueSource)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.<>c__DisplayClass1e`1.<OnBeginGetResult>b__1b()
   at Microsoft.Mashup.Evaluator.EvaluationHost.ReportExceptions(IHostTrace trace, IEngineHost engineHost, IMessageChannel channel, Action action)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetResult[T](IMessageChannel channel, BeginGetResultMessage message, Action`1 action)
   at Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetPreviewValueSource(IMessageChannel channel, BeginGetPreviewValueSourceMessage message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.OnMessageWithUnknownChannel(IMessageChannel baseChannel, MessageWithUnknownChannel messageWithUnknownChannel)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.EvaluationHost.Run()
   at Microsoft.Mashup.Evaluator.SafeThread2.<>c__DisplayClass15.<CreateAction>b__14(Object o)
   at Microsoft.Mashup.Container.EvaluationContainerMain.SafeRun(String[] args)
   at Microsoft.Mashup.Container.EvaluationContainerMain.Main(String[] args)
   --- End of inner exception stack trace ---
   at Microsoft.Mashup.Evaluator.EvaluationHost.OnException(IEngineHost engineHost, IMessageChannel channel, ExceptionMessage message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(IMessageChannel channel, Message message)
   at Microsoft.Mashup.Evaluator.Interface.IMessageChannelExtensions.WaitFor[T](IMessageChannel channel)
   at Microsoft.Mashup.Evaluator.RemotePreviewValueSource.PreviewValueSource.WaitFor(Func`1 condition, Boolean disposing)
   at Microsoft.Mashup.Evaluator.RemotePreviewValueSource.PreviewValueSource.get_TableSource()
   at Microsoft.Mashup.Evaluator.Interface.TracingPreviewValueSource.get_TableSource()
   at Microsoft.Mashup.Host.Document.Analysis.PackageDocumentAnalysisInfo.PackagePartitionAnalysisInfo.SetPreviewValue(EvaluationResult2`1 result, Func`1 getStaleSince, Func`1 getSampled)

Invocation Stack Trace:
   at Microsoft.Mashup.Host.Document.ExceptionExtensions.GetCurrentInvocationStackTrace()
   at Microsoft.Mashup.Client.ClientShared.FeedbackErrorInfo..ctor(String message, Exception exception)
   at Microsoft.PowerBI.Client.Windows.Telemetry.PowerBIUserFeedbackServices.ReportException(IWin32Window activeWindow, WindowsHost windowsHost, FeedbackPackageInfo feedbackPackageInfo, Exception e)
   at Microsoft.Mashup.Client.ClientShared.UnexpectedExceptionHandler.<>c__DisplayClass1.<HandleException>b__0()
   at Microsoft.Mashup.Client.ClientShared.UnexpectedExceptionHandler.HandleException(Exception e)
   at Microsoft.Mashup.Host.Document.ExceptionHandlerExtensions.HandleExceptions(IExceptionHandler exceptionHandler, Action action)
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry tme)
   at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Windows.Forms.Control.InvokeMarshaledCallback(ThreadMethodEntry tme)
   at System.Windows.Forms.Control.InvokeMarshaledCallbacks()
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
   at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
   at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
   at System.Windows.Forms.Form.ShowDialog(IWin32Window owner)
   at Microsoft.Mashup.Client.ClientShared.WindowManager.ShowDialog[T](T form, IWin32Window owner)
   at Microsoft.PowerBI.Client.Program.<>c__DisplayClassb.<Main>b__0()
   at Microsoft.PowerBI.Client.Windows.IExceptionHandlerExtensions.<>c__DisplayClass7.<HandleExceptionsWithNestedTasks>b__6()
   at Microsoft.Mashup.Host.Document.ExceptionHandlerExtensions.HandleExceptions(IExceptionHandler exceptionHandler, Action action)
   at Microsoft.PowerBI.Client.Program.Main(String[] args)

Re: Store historic data after each dataset refresh

v-viig — Fri, 19 Jan 2018 07:44:08 GMT

Could you please sahre the whole R-script that you use?

Ignat Vilesov,

Software Engineer

Microsoft Power BI Custom Visuals

pbicvsupport@microsoft.com

Re: Store historic data after each dataset refresh

Stefan2902 — Thu, 25 Jan 2018 06:30:49 GMT

In the meantime I have figured out the root cause. We're running PBI on a Citrix environment and the R-Client has been installed from a local client. This local client installation of the R-Client doesn't work in PBI. However installing the client directly on the VM image and pointing to the correct R-Client installation solved the problem.

topic Re: Store historic data after each dataset refresh in Developer

Store historic data after each dataset refresh

Re: Store historic data after each dataset refresh

Re: Store historic data after each dataset refresh

Re: Store historic data after each dataset refresh