Forum Discussion

DennesTorres's avatar
DennesTorres
Impactful Individual
2 years ago
Solved

Notebook Logs

Hi,   We can link a workspace with log analytics, I did with mine.   How could I from a notebook generate a log which is recorded in the log analytics linked to the workspace ?   Kind Regards, ...
data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi DennesTorres 
    Thanks for using Fabric Community.
    You can add a code cell in your notebook as follows:

    // Test log4j MDC/NDC
// https://github.com/mspnp/spark-monitoring/issues/28 
import org.apache.log4j.MDC 
val LOGGER_PREFIX = "[Gluten]" 
val logger = org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " +
"com.contoso.LoggerExample") 

logger.info("Hello, info message") 
logger.warn("Hello, warn message") 
logger.error("Hello, error message")

     


    You can validate the log here:


    Hope this helps. Please let me know if you have any further queries.

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi DennesTorres 
Thanks for using Fabric Community.
You can add a code cell in your notebook as follows:

// Test log4j MDC/NDC
// https://github.com/mspnp/spark-monitoring/issues/28 
import org.apache.log4j.MDC 
val LOGGER_PREFIX = "[Gluten]" 
val logger = org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " +
"com.contoso.LoggerExample") 

logger.info("Hello, info message") 
logger.warn("Hello, warn message") 
logger.error("Hello, error message")

 


You can validate the log here:


Hope this helps. Please let me know if you have any further queries.

  • DennesTorres's avatar
    DennesTorres
    Impactful Individual
    2 years ago

     

    Thank you, this seems great!

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi DennesTorres 
    We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
    Thanks

  • DennesTorres's avatar
    DennesTorres
    Impactful Individual
    2 years ago

    Hi, Anonymous ,

     

    This example seems made for Databricks, I couldn't manage to make it work in Fabric.

    First problem is the import: This library doesn't exist and pip install doesn't work for it as well. Is there other way to make this import work ?

    I tried to work around the import in two different way:

    First attempt:

    logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " + "com.contoso.LoggerExample")
     
    Second Attempt: 
    logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getRootLogger()
     
    The workspace was linked to log analytics. The table you exemplified in log analytics doesn't exist. I tried many other tables related to power bi or spark, but none of them registered the log messages.

    What else am I missing?

    Thank you in advance!

    Kind Regards,

    Dennes



    • BlackCurrant's avatar
      BlackCurrant
      Frequent Visitor
      1 year ago

      Hello, I am trying the same, did it work for you? Thanks!

  • vgiatti's avatar
    vgiatti
    Frequent Visitor
    1 year ago

    Until here is fine. The real pain is that the custom tables generated from Log Analytics cannot be exported to Azure Event Hub and consequentially neither in a Fabric EventStream Pipeline.

    See: Log Analytics workspace data export in Azure Monitor - Azure Monitor | Microsoft Learn

     

    Like this, if I want to consume the custom log data in PBI I have to continuosly refresh the semantic model to retrieve fresh data, and everytime you click refresh it queries ALL of the historical logs in Azure Log Analytics.