Forum Discussion

DennesTorres's avatar
DennesTorres
Impactful Individual
2 years ago
Solved

Notebook Logs

Hi,

 

We can link a workspace with log analytics, I did with mine.

 

How could I from a notebook generate a log which is recorded in the log analytics linked to the workspace ?

 

Kind Regards,

 

Dennes

data engineering
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi DennesTorres 
    Thanks for using Fabric Community.
    You can add a code cell in your notebook as follows:

    // Test log4j MDC/NDC
// https://github.com/mspnp/spark-monitoring/issues/28 
import org.apache.log4j.MDC 
val LOGGER_PREFIX = "[Gluten]" 
val logger = org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " +
"com.contoso.LoggerExample") 

logger.info("Hello, info message") 
logger.warn("Hello, warn message") 
logger.error("Hello, error message")

     


    You can validate the log here:


    Hope this helps. Please let me know if you have any further queries.

10 Replies

  • vgiatti's avatar
    vgiatti
    Frequent Visitor
    1 year ago

    DennesTorres is the real king as always.

    The only real pain is that (as of today) the tables are ingested in Azure Log Analytics as "Custom Table (classic)", and these tables cannot be added to an Azure Events Hub, which is the only "make-sense" real-time source to be used in a Fabric Eventstream pipeline.

      • vgiatti's avatar
        vgiatti
        Frequent Visitor
        1 year ago

        Your solution is gold, but still the solution creates Custom Tables into Azure Log Analytics, and that kind of tables as of today cannot be used to fuel Azure Event Hub. Therefore, it is not possible to stream those logs from Azure Log Analytics into a Fabric Lakehouse or Eventhouse.

        The partial solution is to set the log retention period on Azure Log Analytics to 2 years, and query them with KQL queries on PBI or directly on ALA.

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi DennesTorres 
    Thanks for using Fabric Community.
    You can add a code cell in your notebook as follows:

    // Test log4j MDC/NDC
// https://github.com/mspnp/spark-monitoring/issues/28 
import org.apache.log4j.MDC 
val LOGGER_PREFIX = "[Gluten]" 
val logger = org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " +
"com.contoso.LoggerExample") 

logger.info("Hello, info message") 
logger.warn("Hello, warn message") 
logger.error("Hello, error message")

     


    You can validate the log here:


    Hope this helps. Please let me know if you have any further queries.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi DennesTorres 
      We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
      Thanks

    • DennesTorres's avatar
      DennesTorres
      Impactful Individual
      2 years ago

      Hi, Anonymous ,

       

      This example seems made for Databricks, I couldn't manage to make it work in Fabric.

      First problem is the import: This library doesn't exist and pip install doesn't work for it as well. Is there other way to make this import work ?

      I tried to work around the import in two different way:

      First attempt:

      logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getLogger(LOGGER_PREFIX + " " + "com.contoso.LoggerExample")
       
      Second Attempt: 
      logger = spark.sparkContext._jvm.org.apache.log4j.LogManager.getRootLogger()
       
      The workspace was linked to log analytics. The table you exemplified in log analytics doesn't exist. I tried many other tables related to power bi or spark, but none of them registered the log messages.

      What else am I missing?

      Thank you in advance!

      Kind Regards,

      Dennes



      • BlackCurrant's avatar
        BlackCurrant
        Frequent Visitor
        1 year ago

        Hello, I am trying the same, did it work for you? Thanks!