Indy Navarro Website

Plans for this year

Fri, 02 Jan 2026 00:00:00 +0900

I am not really into writing commitments or resolutions for the new year. Because most of the time I ended up changing it in the middle, but I have seen some blog post from the small web sharing their own stuffs, and I thought It could be nice to try once again, so here is my (desired) plan

0.1. 1. Studying some programming languages.

As I wrote previously I have some interest in specific programming languages I don't believe so much in this idea to learn one that could solve all your problems but just choose the best tool for the job (while I understand the idea of having just one or two tools that you need to know to solve most of your problems is a beautiful idea). But nowadays, still use Julia for all the crunching number problems, and I want to develop a higher proficiency with Elisp to master emacs, C for low level programming and maybe Go for any other general programming stuff (I admit that Common Lisp is also an interesting option for the last purpose).

0.2. 2. Develop a Free Software Project

Self-explanatory, but I have interest to develop some software for my own problems, that often I face when I do some scientific computing, because of that I want to try to build solutions for my own, Probably I will use Julia that is the language I have been using the most these days. Let's see if this can produce good fruits.

0.3. 3. Reading intentions

I have several books in my library, from "Thinking, Fast and Slow" to The Bible If is possible to read at least 10 books and 2 written in japanese It would be a nice goal, certainly I could try to read more things but want to have space for other stuffs. And also want to try to write reviews of the books I am reading in the blog, for anyone interested and also for me as a mean to force me to think more deeply about the things I am reading, there are several books out there in Epub and PDF, and with Emacs is easy to take notes and save it for a future use.

0.4. 4. Publishing tutorials about Linux for the hispanic community

This is something I have been thinking for a while, I am in a Signal group where we try to help each other with things related to FOSS, privacy and other related stuffs, people come from different backgrounds and level of understanding about FOSS It is not a surprise that people in the hispanic world have a harder time trying to immerse themselves in the world of technoly when most of the content out there is in english. Well crafted tutorials for specific problems could improve the experience of newcomers.

These tutorials could range from installing linux, basics of the command line, git, having your own website, text editing with vim, emacs, among other utilities. The tutorials could be written in my website and record it so people could watch on Youtube or Peertube.

0.5. 5. Continue mastering emacs.

Time to time I was considering to go back to Vim, when I got tired of editing in Emacs, hoping that now vim and the unix philosophy will open a new door for enjoyment, but this rapidly fall and just decide to go back to Emacs, another case of "the grass is greener on the other side" I guess.

I have been using Emacs for a year, started suffering with the keybindings, learning about org mode, and coding some elisp. Today I use Emacs for almost 80% of my computing experience: emails, news, blogging, coding, reading, etc. But still I am fairly an user that just relay too much in foreign packages when I see there are some nice built-in solutions and I would like to become more proficient with this tools and my capacity to write elisp code.

1. Conclusions

I don't have much to add, let's see how this year is moving and hope everyone has an excellent 2026, stay safe!

Notes about Functional Programing with Julia

Sat, 10 Aug 2024 00:00:00 +0900

I am writing here some general ideas that were taken from some sources like boot.dev about functional programming, many of these sources were written in Python and I just rewrote Julia in most of the cases. Because Julia is a program more suitable for FP I considered a good exercise in the long run to translate the concepts that I am learning about this paradigm.

1. What is Functional Programming

compose functions instead of mutating states,
What you want to happen rather than how you want to happen

1.1. Inmutability

Once the Value is created it can't be changed, this can be easier to debug

1.2. Declarative

Functional aims to be declarative rather than imperative

1.3. Math Style

imperative style

function get_average(nums)
    total = 0
    for num in nums
        total += num
    end
    return total / length(nums)
end

functional style

function get_average(nums)
    return sum(nums) / length(nums)
end

In general to make a bit more functional style, we should avoid loops and mutate any variable

Classes encourage you to think about the world as a hierarchical collection of objects. Objects bundle behavior, data, and state together in a way that draws boundaries between instances of things, like chess pieces on a board.

Functions encourage you to think about the world as a series of data transformations. Functions take data as input and return a transformed output. For example, a function might take the entire state of a chess board and a move as inputs, and return the new state of the board as output.

OOP is not quite the opposite with FP, but the 4 pillars of the first one (abstraction, encapsulation, inheritance and polymorphism) inheritance is the one that can produce changes in classes, so break the rule of inmutability in FP

1.4. Functions are First Class

We can treat functions as values

function add(x,y)
    return x+ y
end

addition = add

println(addition(2,7)

# print 9

1.4.1. Anonymous Functions

Basically functions that doesn't have name, similar like python use lambda functions

function filter_var(df, value)
    return filter!(row -> row.colum != value , df)
end

in the last case row -> row.colum ! value= is an anonymous function

1.4.2. Higher Order Functions

In the case that the programming language threat functions like any other variable, so Functions are first class then we can pass functions as an arguments to other functions.

function square(x)
    return x * x
end

function my_map(func, arg_list)
    result = []
    for i in arg_list
        push!(result, func(i))
    end
    return result
end

squares = my_map(square, [1, 2, 3, 4, 5])
println(squares)
# [1, 4, 9, 16, 25]

In the last case my_map() is a higher order function

Map, Filter and Reduce
Map, filter and reduce are three typical examples of Higher order functions that are quite useful, for a map function you need an iterable (An object capable of returning its members one at a time.) and a function, and apply the function to all the elements of this iterable
```
function say_hello(name)
    return "Hello " * name
end

list_names = ["Chris", "Hector", "Benito"]

map(say_hello, list_names)
# ["Hello Chris, "Hello Hector", "Hello Benito"]
```
Filter was already shown in an example before, but basically takes an iterable, a function and return also an iterable that is a subset of the original.

Finally the Reduce function take same arguments but now it reduce everything to a single value, like the following example
```
function add(sum_so_far, x)
    prinln("sum_so_far: $sum_so_far, x: $x")
    return sum_so_far + x
end

numbers = [1, 2, 3, 4]
sum = reduce(add, numbers)

# sum_so_far: 1, x: 2
# sum_so_far: 3, x: 3
# sum_so_far: 6, x: 4
# 10

println(sum)

# 10
```
This higher order functions allow us to write functions without using loops in some cases avoiding stateful iterations and mutation of variables.

1.5. Pure Functions

Pure functions has to accomplish two properties:

They always return the same value given the same arguments.
Running them causes no side effects

Figure 1: pure

function findMax(nums)
    max_val = -Inf
    for num in nums
        if max_val < num
            max_val = num
        end
    end
    return max_val
end

Let's compare with this other case

# instead of returning a value
# this function modifies a global variable
global_max = -Inf

function findMax(nums)
    global global_max
    for num in nums
        if global_max < num
            global_max = num
        end
    end
end

In the first case we keep a function which clearly define an input and return and output while in the second case we produce a global variable that change the state of this (breaking the rule of inmutability) and does not return anything but our global variable has changed. In summary, pure functions:

Return the same result if given the same input, so they are deterministic (which no randomness is involved in the development of future states of the system.). Also there is the term referentially transparent
Do not change the external state of the program. For example, they do not change any variables outside of their scope.
Do not perform any I/O operation like printing, accessing to data via HTTP or reading files.

1.6. Reference and Value

There are functions that allow you to pass by references, this are mutable, you can see this when appending values in a list. In this case the function has access to the original value. For other side a function that receive variables as values are receiving copy of the original and do not attemt to change the original (inmutability), you can do in Julia using deepcopy(var) to create copies

1.6.1. Pass by Reference Impurity

To avoid side effects we can create copies of the variables inside of a function without changing any variables that is out of the scope (this includes the input of the function)

function remove_format(default_formats, old_format)
    new_formats = deepcopy(default_formats)
    new_formats[old_format] = false
    return new_formats
end

With this we avoid mutating any input or global variable making it easier to debug and test.

1.7. Input and Output

While I/O operations are part of impure functions, these are necessaries (or our program is completely useless) so It tries to use only when is neccesary.

1.8. NO-OP

Functions that does nothing, or better said doesn't return anything, probably are impure functions

function square(x)
    x * x
end

That function doesn't do anything, but also there are functions that perform some side effect:

y = 5
function add_to_y(x)
    global y
    y += x
end

add_to_y(3)
# y = 8

Even the print() function technically has an impure side effect

1.9. Memoization

This is storing a copy of a result a computation so we don't have it to compute it again in the future, it holds a trade-off between memory and speed. This only can be achieved with pure functions.

const fibmem = Dict{Int,Int}()
function fib(n)
    get!(fibmem, n) do
        n < 3 ? 1 : fib(n-1) + fib(n-2)
    end
end

2. Recursion

Function that define itself, for example the classic factorial. This kind of functions are quite useful for unknown tree structure

function factorial_rec(x)
    if x == 0 
        return 1
    else
        return x * factorial_rec(x - 1)
    end
end

julia> factorial_rec(0)
1

julia> factorial_rec(3)
6

A recursive function should have some dangerous edge case that deserve attention:

Requires base case to avoid infinite loops.
Each function call requires a bit of memory, so in long trees structures can cause a stack overflow and will crash your program
In some languages recursion is slow, like python where is even slower than loops. Use of Tail call Optimizations can deal with that

2.1. Function Transformations

Specific type of Higher order functions that receive functions as input and return functions as output, special for some cases of code reusability

function multiply(x, y)
    return x * y
end

function add(x, y)
    return x + y
end

# self_math is a higher order function
# input: a function that takes two arguments and returns a value
# output: a new function that takes one argument and returns a value
function self_math(math_func)
    function inner_func(x)
        return math_func(x, x)
    end
    return inner_func
end

square_func = self_math(multiply)
double_func = self_math(add)

println(square_func(5))
# prints 25

println(double_func(5))
# prints 10

2.2. Closures

A closure is a function that references variables from outside its own function body. The function definition and its environment are bundled together into a single entity so a closure can change the value outside its body

Figure 2: closure

julia> function make_adder(amount)
           function add(x)
               return x + amount
           end
       end;

julia> add_one = make_adder(1);

julia> add_two = make_adder(2);

julia> 10 |> add_one
11

julia> 10 |> add_two
12

In the case of Julia, generate global variables can cause Type Instability and there are some discussions about avoiding closures when performance is required, However that doesn't mean that using closures should be avoided completely, a lot of discussions are here also interesting content here

Naturally if a function can change a a non local variable then is not a pure function, so many cases closures are not pure functions because they can mutate outside of their scope and have side effects.

Notice that also there are concept of Decorators in some languages like Python, that are just syntactic sugar for higher order functions

2.3. Currying

Function currying is a specific kind of function transformation where we translate a single function that accepts multiple arguments into multiple functions that each accept a single argument.

Figure 3: currying

This is a normal function without currying

function sum(a,b)
    return a+b
end

With currying

function sum(a)
    function inner_sum(b)
        return a + b
    end
    return inner_sum
end

With this option now we can return a function as a value (inner_sum) and change it's signature to make it conform to specific parameter

2.4. Wrapping up

These are just basic ideas about functional programming, there are more concepts to deal with, but at least here is an starting point for people like me who is not a cs person…

Consider to use webrings for your site

Sun, 17 Mar 2024 00:00:00 +0900

"Hey you Out there beyond the wall Breaking bottles in the hall Can you help me?" — Roger Waters

Do not make a wall for your website, that place that in case of any visitor want to come and read any article or all your blog, feel that there is nothing more to see an return wherever he came. Instead, give it more where he can continue his fly.

Social media do this all the time, you just finished watching a video on YouTube and immediately the platform will recommend you another videos. Do the same, but for better reasons.

Create a post in your blog with the interesting websites you enjoy, link other websites when you want to cite their ideas, add at the end of your articles the latest posts of some blogs that could complement (or contrast) your ideas. For the last one, you can use a very simple tool like openring, which is the tool I use for adding a webring in this site.

If you want to use this tool, please make sure you have installed the Go programming language, then open your terminal and write:

$>git clone https://git.sr.ht/~sircmpwn/openring
$>cd openring
$>go build openring.go 
$>./openring \
  -s https://bkamins.github.io/feed.xml \
  -s https://jarbus.net/index.xml \
  -s https://www.paltmeyer.com/blog/index.xml \
  < in.html \
  > out.html

Replace these websites with the ones you are interested to show to the public and find their respective rss feeds. Notice that in.html is a template, so you can customize as you want to get a good look in your own website, for other side out.html is the template to feed it with the latest articles of the websites you chose, and that is the one to insert in your blog.

Here I am using Hugo as a website, so I just left the out.html in the partial folder and then modify the post layout configuration with something like:

{{ partial out.html . }}

And That's all the effort you have to do to make the web a bit better. We don't have to be isolated, together we stand, divided we fall.

Sway + Vim + Tmux + Julia: 3 Data Science Workflows

Mon, 22 Jul 2024 00:00:00 +0900

Occasionally, I reviewed my workflow with different tools and saw if that was comfortable enough for the development process. When I was using Vim, someone pointed out that they were using it for data science, which caught my attention. I tried to experiment with different ways I could use the tool. Using Vim as a tool to replace Jupyter Notebooks first took a lot of work, mainly because I missed the experience of seeing all the plots and data in just one website and being able to scroll all around. In my experience with Julia for a data science project, there are 3 ways you can achieve a good experience with different trade-offs.

0.0.1. Basic ingredients.

A Tiling Window Manager, in my case, I use Sway
Vim
Tmux
Julia or any other scientific computing language

1. Double Pane

Figure 1: vimtmux

This is the one I read most of the time on Reddit. Here, you will need to install Vim-Slime or a similar "send-to-REPL" plugin. In this scenario, you will enjoy the feature of sending snippets of code to your REPL. You only need to activate Tmux, open your .jl file on one side and the REPL on the other using Tmux panes, and that's it.

The pro is that you can run all your files or experiment with different snippets. The cons of this workflow are that you will be constrained by the space of your screen when it comes to seeing your plots because it will use half of the screen when it displays. The double pane will be shrunk, so you cannot clearly see the code and the REPL. To mitigate this, you could use UnicodePlots to see your data and plots only in the REPL without emergent windows.

I used this as a standard for a while, but it didn't click on me when working with plots in this workflow.

2. One Pane

You can achieve this without using Tmux (but it is still convenient to display different sessions). Julia has a really powerful REPL, and the way you can change to Shell mode deserves more attention. Many people will not understand why that is important, but Julia's key is to stay with the REPL activated as much as you can. If you need to shut down it and do other things, then you lose a lot of the power that Julia holds.

Some people dislike using REPL-based languages because they force you to work with workflows that deal with code snippets instead of running all your programs at once. But this is not true at all.

Similarly, you run julia my script.jl in the terminal and then open the file with Vim to keep editing, you can use the REPL to change to the Shell mode, modify your script and then come back to the Julia mode and run your script using include("myscript.jl").

In particular, you can work very well with this work and keep your workspace cleaner. However, if you are working with a script that creates several variables for data cleaning and plots, you will be annoyed to try to remember all of them; at least, that happened to me. When I close the script and run it, sometimes I want to check some results stored in specific variables and call them again, but not always remember them all. You can mitigate part of this by using a main function, saving your plots, and processing the data.

3. 2.5 views panels

This try to replicate the experience on RStudio or VSCode you will mainly will deal with 2 panels, one with your code, and for other side The REPL, these terminals are splitted, so you are not using Tmux to open panes as in the first case.

Figure 2: views

Here is where your Tiling Windows Manager will shine in your workflow. If you set up correctly, a new emergent window will appear on your screen every time you run your script and display the plots you want to see. You can easily explore it, zoom in and out, close it, and return to work with your script. I am having a good time using sxiv from suckless to display plots. If you need to see the plots again, here is where the REPL and the Shell mode can help you; if you saved it, you could open it up using tools like Sxiv to explore your plot again.

The positive thing about this style is that you can have everything displayed in a comfortable way, see the code and the REPL clearly, and display your figures in a noninvasive way when you need to explore your figures. The bad thing about it, again this disminished the utilities of Tmux and also made you dependent of your tiling windows manager, not all work in the same way; if you use Sway or dwm, for example, you will have different experiences given the basic set up, so you will have to change the configuration to set your own needs.

4. Conclusion

Finally, I want to add that there is still room for improvement, and I would like to see other people's workflows and how they deal with their data science projects. These days, I am more inclined to use the last option for this kind of project, and the first one I found more useful for software engineering, like a project where interactivity is not a central part of the process.

How to scrape data with Python using selenium and Pandas

Thu, 15 Dec 2022 00:00:00 +0900

1. Introduction

In this tutorial, I will dedicate myself to explaining how web scraping can be done from a platform where a dynamic interaction of the web application is required, this is quite useful when obtaining data from different links within the platform and where it is necessary a management scheme of the front-end components to carry it out.

Here there are mainly two essential libraries, the first is selenium which corresponds to a framework that operates for multiple languages and serves to automate and control the browser, while Pandas for data manipulation will allow us to read data tables directly.

Many times, the beautiful soup library is used to extract html elements from the web, but as we will see, it is not necessary to do so in this case.

For this example, I am going to use the chilean dairy production platform, this platform is used to obtain information on the production of products dairy products from different factories nationwide.

2. Requirements

To start, you must have Python installed. In my case, I am using version 3.9, you also have to have your browser (Mozilla or Chrome) secured. In this project, I will use the chrome one, but the codes should be similar to the one we are using here, then to work with selenium, you have to download the [executable](https://chromedriver.chromium.org/downloads) that corresponds to your browser and its respective version

If you use pip you can install it using:

pip install -U selenium

The import the libraries

from selenium import webdriver
import pandas as pd
import lxml
from selenium.webdriver.support.ui import Select
import sys
import time

Once you have imported the corresponding libraries, we will perform the first test with the chromedriver.exe (the one you downloaded from the selenium portal). For simplicity, I recommend having it in the same directory as this scrapper.

driver = webdriver.Chrome('/Your/path/to/the/project/chromedriver')
driver.get("http://aplicativos.odepa.cl/recepcion-industria-lactea.do")

This should allow the web page to be opened from the Chrome browser, the driver variable that we have assigned the chromedriver will drive the states of our browser. We can now add this snippet code

time.sleep(5)
driver.quit()

With this, we add a timeout of 5 seconds, and with driver.quit() we close the browser. The reason for adding waiting times is that while we have to operate within the browser, either due to internet connections or latency of the web platform, we will therefore have to wait for the elements we need to be available.

It is time to see how we can start interacting with the web page elements. For example, if we want to click on certain features, what we have to do is right-click on the component on the web page, place inspect and then recognize the element and how we can call it according to how it is identified, this can be by id, name, XPath, etc. I often use the XPath, which you can copy and paste into your code.

#Select elements
driver.find_element_by_id('tipoConsulta2').click()
driver.find_element_by_id('filterByRegionOrPlanta2').click()
driver.find_element_by_id('filterByRegionOrPlanta2').click()

#Extract the list of years
driver.find_element_by_xpath('//*[@id="divFechaDetalleMensual"]/img').click()
driver.find_element_by_xpath('//*[@id="ui-datepicker-div"]/div[1]/div/select').click()
years = driver.find_elements_by_tag_name("option")

Here what we have done is open the web page and make the necessary selections and filters to access the data, we end up creating a list called years, where we will have all the years available in this web application.

Now with this, we can get the elements. Using the following code.

  list_years = []
for year in years:
    list_years.append(year.get_attribute('value'))

#here I added a filter by year which is optional (you can delete it)
list_years = [element for element in list_years if element != '' and int(element)> 2000]

Now we will obtain the list of elements of all the years to be able to iterate. Then if we want to get the plants, we can use the following:

  #Extract all the factory names:
plantasposibles=driver.find_element_by_id('planta')
plantasposibles=plantasposibles.find_elements_by_tag_name("option")
valoresplantas=[]
nombresplantas=[]

for option in plantasposibles:
    valoresplantas.append(option.get_attribute("value"))
    nombresplantas.append(option.get_attribute("text"))

We locate the dropdown that corresponds to the list of available plants, with this, we take the elements and build the list of plants. This will allow us to perform the following iteration:

tabla=pd.DataFrame() #Here we create the dataframe

driver.find_element_by_xpath('//*[@id="divFechaDetalleMensual"]/img').click()

for lastyear in list_years:
    for i in range(1,len(valoresplantas)):
    ...

We need to start controlling the options and release the report with the data. From there, we perform reading and data extraction, this is where Pandas shines. If we remember the last double loop, what should go inside is the following.

  #Select options
driver.execute_script("document.getElementById('planta').value="+ valoresplantas[i])
driver.find_element_by_xpath("//*[@id='divFechaDetalleMensual']/img").click()
time.sleep(1)
select=Select(driver.find_element_by_xpath("//*[@id='ui-datepicker-div']/div[1]/div/select"))
select.select_by_visible_text(str(lastyear))        
driver.find_element_by_xpath("//*[@id='ui-datepicker-div']/div[2]/button").click()
driver.find_element_by_id('fechaDetalleMensual').send_keys(lastyear)
timeout=15
driver.find_element_by_id('btnVerInforme').click()
timeout=20

############################## PANDAS #######################################

prueba_html=driver.page_source
df = pd.read_html(prueba_html, flavor='html5lib')[0]
df=df.drop(df.columns[14:397],axis=1)
df=df.drop(df.index[0:8],axis=0)
df=df.drop(df.index[1],axis=0)
df=df.drop(df.index[8:9],axis=0)
df['Year']=lastyear
df['Factory_Name']=nombresplantas[i]
tabla=pd.concat([tabla,df])

In case it fails, which is typical when working in selenium, the try/catch options are the best to handle exceptions intelligently. Obviously, it depends a lot on the case and the nature of the project on how to use them, but here I just proceeded to close the application and operate again where it was. To summarize this point, the double loop would look like this:

for lastyear in list_years:
  for i in range(1,len(valoresplantas)):
      try: 
          driver.execute_script("document.getElementById('planta').value="+ valoresplantas[i])
          driver.find_element_by_xpath("//*[@id='divFechaDetalleMensual']/img").click()
          time.sleep(1)
          select=Select(driver.find_element_by_xpath("//*[@id='ui-datepicker-div']/div[1]/div/select"))
          select.select_by_visible_text(str(lastyear))        
          driver.find_element_by_xpath("//*[@id='ui-datepicker-div']/div[2]/button").click()
          driver.find_element_by_id('fechaDetalleMensual').send_keys(lastyear)
          timeout=15
          driver.find_element_by_id('btnVerInforme').click()
          timeout=20


          prueba_html=driver.page_source
          df = pd.read_html(prueba_html, flavor='html5lib')[0]
          df=df.drop(df.columns[14:397],axis=1)
          df=df.drop(df.index[0:8],axis=0)
          df=df.drop(df.index[1],axis=0)
          df=df.drop(df.index[8:9],axis=0)
          df['Year']=lastyear
          df['Factory_Name']=nombresplantas[i]
          tabla=pd.concat([tabla,df])

      except:

          #If fail close and open up the window again
          #driver.quit()
          time.sleep(5)
          driver.get("http://aplicativos.odepa.cl/recepcion-industria-lactea.do")
          time.sleep(5)
          driver.find_element_by_id('tipoConsulta2').click()
          driver.find_element_by_id('filterByRegionOrPlanta2').click()
          driver.find_element_by_id('filterByRegionOrPlanta2').click()

3. Final steps with Pandas

With this, we would be finishing the process, the only thing left is to integrate the final data with some extra elements and save the dataframe. We will ensure that each component is integrated with its period since the months are in columns, so we will make a single column that contains them.

tabla=tabla[['Year', 'Factory_Name', 'Product', 'Unit','Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec']]

lista=range(len(tabla.index))
tabla.index=lista

tablafinal=pd.DataFrame()
tablaparcial=tabla.drop(tabla.columns[4:],axis=1)

for month in tabla.columns[4:len(tabla.columns)]:

    tablaparcial['Month']=month
    tablaparcial['Quantity']=tabla[month]
    tablafinal=pd.concat([tablafinal,tablaparcial])

tablafinal.to_csv("data.csv", index = False)

Finally, we can do the extraction and a simple preprocessing to leave them more prepared for some analysis or save them to a database.

4. Conclusions

In this project, we show how we can perform scrapping using selenium and pandas, this of course, can be done thanks to the pandas tools to extract data from HTML, simplifying the extraction. Selenium is an excellent tool to carry out this automation and test web pages, so I recommend it for the design of web apps, for example, failures in the results or scenarios where there are possible bugs.

Leaving Elfeed for Gnus

Wed, 13 May 2026 00:00:00 +0900

A few weeks ago I was playing around with Gnus, I don't remember Why I stop using it but I just switched back to Elfeed, and before that I also spent my time with Newsticker, I wasn't convinced completely about the last one, so seriously start using other unix tools for reading news, like Newsraft.

Just few weeks ago the Elfeed's author announced that finally stop using Emacs in favor of Vim, and he vibecode his own solution called Elfeed2. Obviously this situation put at risk the original project in case that no one wants to become the maintainer. So I took the opportunity to revisit the built-in tools in Emacs and try Gnus again, because It gave me a better experience compared with Newsticker. For this reason, I have to say that the Emacs people who tend to prefer the built-in solutions and just hack around it, got it right.

After a couple of tutorials from Protesilaos and Amin Bandali, I could set up successfully Gnus while understanding the basic keys for navigation. Gnus is big, there is no doubt about that, in fact I still don't read all the documentation, but you need very little to start with it.

The last piece in the puzzle was to be able to set up mpv for specific use cases (like watching Youtube videos), I was searching a way to do this with some Gnus features, but later I just realize that with just changing the default browser, you can easily execute mpv when there are some extensions like "Youtube.com…"

1. Basic Tutorial

So what I recommend is following some servers, while Gnus has his own functions to extract rss (nnrss), I just recommend you to connect to Gwene.org and Feedbase.org via nntp to follow your blogs, mailing list and Youtube channels.

(use-package gnus
:ensure nil
:config

(setq gnus-select-method '(nnnil ""))
(setq gnus-secondary-select-methods
      '(
          ;; Connecting to gwene
          (nntp "news.gwene.org")
          ;; Connecting to Feedbase
          (nntp "feedbase"
                (nntp-open-connection-function nntp-open-tls-stream) ; feedbase does not do STARTTLS (yet?)
                (nntp-port-number 563) ; nntps
                (nntp-address "feedbase.org"))
          ;; Here can come another connections like your emails for example
)

With this basic setup, you can start Gnus and you should be able to connect to these servers, obviously nothing will appear because you didn't subscribe to anything yet, but if you press ^ you should be able to see all the servers you are connected to, select gwene, it will open all the groups in that server find your blogs, mailing lists and Youtube feeds, if you are missing some just add the rss to the respective websites (gwene.org) and in minutes, the new rss will appear on the list, you can subscribe to the group by putting the cursor on the group and pressing u.

After You subscribed to few groups close the servers with q and you should come back to Gnus Groups, you can update pressing g or just quick with q and enter again to Gnus, You should see the new groups you subscribed.

Another nice feature of Gnus is that you can create topics, this is just a simple way to sort your groups so it become easier to navigate through them. Press T-n to create a topic, and to move specific group to that topic just press T-m and then select the topic to send that group to it.

Finally, you want to read the news, select the group if it has too many articles Gnus will prompt you how many you wants to get, select 10 for example and then it will open the latest 10 articles, you can press Enter to open the article, n and p to move to the next and previous article unread, Space to reading through the article and q to close the Group and comeback to the main Gnus default. If you want to mark all the articles in the group as read, just press c and select y.

If you want to read the article from another way, let's say Youtube from mpv and blogs from eww, I recommend you to use a code I steal from Joshua:

  (defun my-browse-url-mpv (url &rest _args)
  "Open URL in mpv."
  (start-process "mpv" nil "mpv" url))

(defun my-browse-url-pdf (url &rest _args)
  "Fetch remote PDF and open in pdf-tools within Emacs."
  (let ((tmp (make-temp-file "emacs-pdf-" nil ".pdf")))
    (url-copy-file url tmp t)
    (find-file-other-window tmp)
    (pdf-view-mode)))

(setq browse-url-handlers
      '(("\\(youtube\\.com\\|youtu\\.be\\|vimeo\\.com\\|twitch\\.tv\\)" . my-browse-url-mpv)
        ("\\.mp4$" . my-browse-url-mpv)
        ("\\.pdf$" . my-browse-url-pdf)
        ("^gemini://" . elpher-browse-url-elpher)
        ("^gopher://" . elpher-browse-url-elpher)
        ("." . eww-browse-url)))

This allow me to open articles with eww, and youtube and other websites with mpv. I am sure there is a better way to handle the urls with Gnus but for now this solve my basic use.

2. Conclusions

Gnus is huge, and many people prefer the simplicity of Elfeed to access to news and blogs, but while that is true, I don't think you need to know that much about Gnus in order to consume information, surprisingly enough, Gnus has been a nice experience for reading news and emails. and I think We should appreciate that we have such a solid tool inside Emacs.

Status update november 2025

Sat, 06 Dec 2025 00:00:00 +0900

Well, this is my first status update in the blog, before I was writing some things based on a specific topic, as today I have many things happening but I don't have the interest to go so much into details to explain what I am doing or thinking. So at least I want to give a time to describe briefly these things.

1. Dreaming in Lisp

In the past few months I have been exposed to the Lisp family language, first with Emacs Lisp (Elisp) and my curiosity for the language has increased, while Elisp only can be used inside Emacs, I was checking a more general programming language, from a long review it seems that Clojure and Common-Lisp are the most popular programming languages for that purpose.

I am now getting some expositions with CL through Exercism, and I would like to participate in the Advent of Code 2025 if I have some time these days. But overall I like the experience, certainly coming from Julia, it has been a pleasant experience to use Emacs and Slime with SBCL, while I like Julia, I feel like dealing with compilation time and the lack of mature tooling outside of VSCode get me exhausted sometimes.

Also I gave a time to use StumpWM, a window manager written in Common Lisp, it feels snappy, and the way that you can connect Emacs to the REPL and start making changes on the fly is a much more productive workflow than the suckless experience of compiling and restarting your software. Also the way that you can create and rename Groups and search by name has been a saner experience than workspaces based on numbers in my opinion.

2. Change to Graphene-OS

Few weeks ago I started using a Pixel 8 with Graphene OS, certainly in this experience I was hesitant to move to acquire a phone that can be impossible to unlock due to carrier restrictions, and you could ended up with a Google Phone. So I approach this with serious caution.

Fortunately, everything ended up well, and I could flash the new OS in the phone, install a sandboxed Google Play Store and install some apps that I need for daily basis, and the rest fill with only FOSS.

Overall the experience has been better than I expected, I created multiple users and when I need to move to the Google services I just move between users, It has been a peace of mind comparing with using a Huawei phone, considering that you have 2 bigh tech sneaky around your data.

3. Self Hosting my XMPP server

For a while I was considering to self-host a chat server for family and friends, and you have mainly 2 options XMPP or Matrix, I try to see the installation process of both using the most common protocol implementations (Prosody and Synapse respectively), and I really want to like Matrix, but I just feel Prosody was far more straightforward and easier to install, It was in the Debian Repository so you could just type sudo apt install prosody in the terminal, while I think also Matrix works out of the box and for some reasons I see most people and FOSS projects are using it.

Anyway, the big challenge is to bring people to use this thing, and that is the hardest part, people are so attached to use Whatsapp and other apps, that is hard to convince them to move to something that they will use it only for talking to one contact or one group probably. So still there is work to do.

Enable several features, so people will be less reluctant to change if they compare with the apps they were using (video calls, image upload, chat groups, etc)
Wait for an event that compromise these apps in terms of security and privacy. So if your people want to move to another app, the XMPP server will be ready to receive new users.

4. Thinking to Start Exploring Gentoo

Last weeks I was seriously considering to try a new Linux Distro, I have been using Debian for quite a while, as a server and desktop, but I am considering to use something different for the later, one of the main reasons is to find a Distro that is not using systemd as an init system, there are political and technical reasons to avoid it. For other side, while there are some distros that are branches from the main ones which give you alternative init systems like Devuan and Artix, I don't like to use distros that are based on hacks of the original distribution. So that's why I am thinking to try Gentoo as my possible next distro.

5. Expanding skills on Programming

Finally, I have been a data analyst for a while, my main jobs always has been related with deal with data, I am ok with that, but I am not really into data in my free time. I believe that is good to spend my time in skills that not depends on what the market want from you. The talents should belong to you and only you, data science and all these things are mainly skills that are good for your employee, and I think is important to departure sometimes from that. Learning Linux or Emacs has made me more free than learning something like Power BI or Excel.

That's why I was considering to learn more about programming not only for the niche I was used to do professionally but also for building things, that help me in my own business.

How to set up mail in emacs with mu4e

Wed, 10 Sep 2025 00:00:00 +0900

I was considering for a while to use mail in emacs, I didn't have the time to work on that, and I didn't find a tutorial up to date to just copy and paste, there are some good sources and documentation, but the implementation was not that straightforward, it took me a couple of days to find why It didn't work out my implementations.

Figure 1: mail

But here is the step by step. I have been using Migadu for a while, and It works quite well

1. Software Installation

It will require to install:
mu4e
isync/mbsync
msmtp

sudo apt install isync

Create a new file ~/.mbsyncrc

     
IMAPAccount migadu
Host imap.migadu.com
User mail@mymail.xyz
PassCmd "pass mail/my_mail_password"
TLSType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt
#AuthMechs Login
Port 993

IMAPStore migadu-remote
Account migadu

MaildirStore migadu-local
Path ~/Mail/migadu/
Inbox ~/Mail/migadu/INBOX/
Trash ~/Mail/migadu/Trash/
SubFolders Verbatim

Channel migadu
Far :migadu-remote:
Near :migadu-local:
Patterns *
CopyArrivalDate yes
Create Both
Sync All

SyncState *

Then, make sure to run in the terminal the following statement:

mu init --maildir=DIR --my-address=ADDRESS

then run:

mu index

2. Set up Emacs

Go to your init.el and write this snippet

(use-package mu4e
:ensure nil
;; :load-path "/usr/share/emacs/site-lisp/mu4e/"
:defer 20 ; Wait until 20 seconds after startup
:config

;; This is set to 't' to avoid mail syncing issues when using mbsync
(setq mu4e-change-filenames-when-moving t)
(setq mail-user-agent 'mu4e-user-agent)

;; Refresh mail using isync every 10 minutes
;;(setq mu4e-update-interval (* 10 60))
;;(setq mu4e-get-mail-command "mbsync -a")
(setq mu4e-get-mail-command (concat (executable-find "mbsync") " -a"))
;; how often to call it in seconds:
(setq mu4e-update-interval 300)

(setq mu4e-maildir "~/Mail/migadu")

(setq mu4e-drafts-folder "/Drafts")
(setq mu4e-sent-folder   "/Sent")
(setq mu4e-refile-folder "/All")
(setq mu4e-trash-folder  "/Trash")
(setq mu4e-junk-folder  "/Junk")
(setq mu4e-others-folder  "/Others")
(setq mu4e-maildir-shortcuts
  '(("/INBOX" . ?i)
      ("/Sent" . ?s)
      ("/Trash" . ?t)
      ("/Drafts" . ?d)
      ("/Junk" . ?a)
  ("/Others" . ?o)
  ))
(mu4e t))

With this you can run mu4e inside emacs and you should see how your system is pulling up your emails. But notice is not ready to send emails yet, we need to consider an extra customization.

3. Sending emails with msmtp

Make sure to install msmtp (sudo apt install ) and create a file ~/.msmtprc, and write the following

defaults

auth on

tls on
 
port 587
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        ~/.msmtp.log


account personal
host smtp.migadu.com
from mail@mymail.xyz
user mail@mymail.xyz
passwordeval pass mail/mail@my_mail_password | head -1


account default : personal

Finally write in your emacs 'init.el' this.

  (setq
   message-send-mail-function   'smtpmail-send-it
   smtpmail-default-smtp-server "smtp.migadu.com"
   smtpmail-smtp-server         "smtp.migadu.com"
   smtpmail-local-domain        "migadu.com"
   )

(setq sendmail-program "/usr/bin/msmtp")
(setq message-sendmail-extra-arguments '("--read-envelope-from"))
(setq message-sendmail-f-is-evil t)
(setq message-send-mail-function 'message-send-mail-with-sendmail)

Then you should ready to go, enjoy the spam…

4. Resources

Trying to convert Emacs in a language learning tool

Sat, 14 Feb 2026 00:00:00 +0900

These days I have been considering seriously to keep improving my Japanese while using most of the Emacs tools and ecosystem, the main reason for that is because I just want to stay focus as much as I can, avoiding context switching and having most of my sources on my pc using the tool I like it.

For one side, I use elfeed to suscribe to some Japanese Youtube channels and blogs, I play the videos using mpv. also there is nov.el and PDF tools for reading books. I get my books from different sources, There is a site called Aozora Bunko for example, that is a huge library of books that you can read for free online or local.

Org mode is a nice way to take notes, schedule tasks, among others and combined with Org-drill allow me to create flashcards and practice recognition and recall. I use Org-capture and abbrev-mode to create new vocabulary, grammar or expressions on the fly with some templates I made, using Magit You can track all the changes on these files, this is important because in case that you break your desk, it's easier to reverse any undesired change. Furthermore, I can easily record this new things. Given that I have to write in the language Emacs has its own japanese input method, while it's not perfect, it gets the job done, I can just press C-\ to switch between japanese and english or spanish.

Other important thing I have to add is a dictionary, quick-sdcv is a package that wrap the terminal dictionary sdcv, this allow me with some keybindings to look up words quickly, if I am reading an epub file or news I can just put the cursor on the word and display the meaning in other buffer, I can copy and capture the vocabulary in an org file, or anything else. And there are some IRC groups in Libera.chat where you can chat in japanese, but I do not use it that much to be honest.

Finally, one thing I didn't try yet, is using a LLM inside Emacs, I just use sometimes ChatGPT in the browser with some interesting results, it can be a useful tool, but I don't want it to become an essential part of my workflow. However, I still not completely closed to incorporate this tool inside my workflow.

Okay, but was it necessary to bring all these things into Emacs? Well, maybe not. But if it helps me keep improving my Japanese and enjoying the experience, then it's worth the workaround.

Fundamentals: The Bill Evans Aproach

Wed, 20 Mar 2024 00:00:00 +0900

A long time ago, I was really into music. I was learning guitar and practicing in my free time as much as I could. I also like jazz. I am not an expert on the style, but I like the freedom and virtuosity of this kind of music, and I listened to some of its exponents. One of them was Bill Evans.

Bill Evans was a genius pianist in the jazz scene, with his authentic and complex style, he could easily partner his reputation with other giant musicians like Miles Davis and John Coltrane, and if you have heard something about this music, you will probably know that the most popular album of all times "Kind of Blue", Evans was one of the musicians that took part of this masterpiece. Evans's piano style was considered disciplined to the level where he was an excellent session musician and a brave soloist, making him a guy of respect even among the kings of this music style.

Fortunately, there is on YouTube a very intriguing interview he gave about improvisation, which you can watch (even if you are not a jazz musician, I believe you could enjoy it); one of the things to highlight is his idea that fundamentals are the key, and you should study it properly. This could sound like trivial advice. Still, the reflections he left are interesting because, in his own point of view, your own style will be developed progressively with the advocacy of solid pillars (the theory you dominate). It would be a mistake to focus on learning style.

In music, you can distinguish the style of some musicians who have decided to learn from the style of other musicians. For example, no one could deny that Jimi Hendrix really made a significant impact on the next generations of guitar players, and many of them embraced his style and later developed their own, using Hendrix and other musicians as a framework, but some of them still sounds like those early guitar heroes from the 60s or 70s. That's why, on the other side, Bill Evans was so unique in his style, he didn't spend the time to learn how to play like his predecessors, he spent a considerable amount of time trying to learn about the dynamics and the theory behind the music and little by little developed his complex style based on his analytic mindset and experience. I read in an old guitar book a long time ago, that we could find two kinds of musicians: the ones that are heavily based on theory and others heavily based on experienced patterns. Obviously, every musician is a mix of these 2 "paradigms", but Bill Evans never denied the importance of the first one.

Despite the enormous importance and advantages of learning the fundamentals of every discipline, there is always a trend to avoid the fundamentals, just because it is not fun or because you don't "need" that in the job. So we decide to become pragmatic, and we want to do the cool things: the people who want to learn a (human) language don't want to study the grammar rules and decide to learn just by input content, people who want to learn an instrument avoid practicing scales but decided to learn some pieces of songs they liked, programmers who do not understand well how computers work ignore the proper study of data structures and algorithms but just one to build things.

I am not against simplicity or learning in a fun way; I also want to enjoy the learning process and have several times fallen heavily into this approach, let's call it the shallow approach. This approach gives us the sensation of dynamism that we will not find in books or an academic environment. Conversely, the market always pays attention to the trends, and the shallow content creation has become an essential part of an ambiguous learning industry. If you want to learn the fundamentals of your discipline, maybe you need a few books that cover most of the relevant things you should know, but anyway, we decide to immerse ourselves in shallow content and activities. As you can see, it is relatively easy, but you must learn to study essential things properly.

If we want to become a little bit closer to Bill Evans's style in the discipline advocated, that's the price we have to pay.

Self-hosting my own git server

Thu, 23 Oct 2025 00:00:00 +0900

Few months ago I just decided to get my own vps for self hosting few things, I am not a sysadmin by trade, so it's a bit of a challenge to deal with a linux server, but the reward is huge, I can't think in a more useful thing that you can learn from the tech world than the skill of managing servers for your own services (and maybe family and friends if you think you are capable of).

The first thing I self host was this website, using Nginx and letsEncrypt help me to show this website to the world, but also later I decided to self-host my library with Calibre, and try few other things like Pleroma (when I was in the Fediverse), Searxng, and now I was considering to host my own git server.

As a wanna-be tech independent guy I found very little purpose to have a full feature git client like Github, most of my code it's just means for reproducing calculations so people are not going to do pull request to add new features to code that is for data analysis generally. But if it happens, welcome!.

That's why I just set up Cgit, for the purpose, which is fairly simple, is in the Debian repository and has a very narrow scope, so if someone one send a patch, it has to be by email. The procedure for installation was quite simple, most of what you need you can read here

Hello, World!

Sun, 16 Oct 2022 00:00:00 +0900

1. Introduction

The purpose of starting this blog is to create a space where I can write about topics that interest me related to technology, the world of quantitative methods for decision making, programming, among others.

But why create a blog, if there are so many spaces out there that are more convenient than building one? The truth is, I wanted to have a more independent space and not too comfortable on third parties (nor benefit them), although there are spaces like Twitter (which seems to be in greater decline every day) or Medium, I simply thought that the best thing I could do is to have a personal space, and given that the dream of owning a house seems to be increasingly distant for my generation, well at least I can settle for having a blog, right?

2. Inspirations

The truth is, I've had some inspirations from different blogs out there, and they do very interesting work and opened my desire to have my own space, so I'll mention them here and I hope that you can research on your own and take the time to read their proposals

And several more…

3. What to expect from this blog?

Honestly, I don't have a clear direction yet and at least in the short term, I don't expect to make money with this, for this reason, I'm not going to track visits to this blog.

But some of the topics I will or wish to publish are related to:

Theoretical and applied concepts on statistics, optimization or Machine Learning especially to the world of decision making, as well as related to computer science or software engineering in general.
On the use of Julia and Python to face specific problems of the topics mentioned above. I may be adding other languages that interest me, like R, C/C++ or Rust.
Analysis of certain data that interest me
Open source and technology
Opinion and general reflections on social and life issues

I don't pretend to be an expert voice on any of these topics, but I do hope to learn a lot within this activity of reflecting and writing about these topics.

4. How can you support me?

I don't expect much from this blog other than for my own satisfaction. But in case you want to connect and/or talk about these topics, you can use Mastodon channels or simply my email for that purpose (I'm not a big fan of traditional social networks, to be honest)

In the end, I hope we can have fun and let's connect if we are interested in similar topics

Creating your own blog with Julia and Franklin

Wed, 16 Aug 2023 00:00:00 +0900

In this post, we are going to discuss how to build your own blog with Julia and Franklin.jl, a popular static site generator among Julia users who create their own blogs or even build websites for tutorials. I hope that if you are reading this entry and you don't have your own space, it can motivate you to build your own website.

1. Some Reasons to Create Your Own Blog

Blogs may sound old-fashioned, something created by people who are still living in the 90s, typing with passion about the political system while listening to Soundgarden in the background and drinking some kind of cheap beer… or programmers. And because if you are reading this content, you're probably at least the second one, you should consider that having a blog is a nice way to:

Track your progress in your field
Generate content that can be useful for somebody else
Help the open-source community with diffusion, tutorials, etc.
Create your own space and adapt it you your needs
Build your personal brand and help you to find a job

But why Franklin? Franklin is one of the most popular libraries for this purpose in Julia. It offers seamless integration with running Julia scripts so you can use julia for demostrations in your blog this coud be harder with other static site generators. If you only want to create basic entries with some code and images, perhaps Franklin.jl might not be that different from Hugo or Jekyll.

2. Installation

The first step is to create a folder where you will save your project. Once you are ready, open the Julia REPL in the location where the folder should be. When it's ready, type ] to activate the package manager and then type:

(@v1.9) pkg> add Franklin

then, return to the Julia Repl and import the library:

julia> using Franklin

Remember to make sure you have successfully installed the Franklin library before trying to import it.

3. First Steps

To create your website, you can choose one of the templates available. In my case, I just used the basic one, but if you have a different preference, feel free to go ahead; they all follow similar structures. You can also import another template that you like more and adapt it to your website. Please read the documentation for instructions on how to do this.

3.1. Selecting a template

Once you have decided your template, type in the REPL the next instruction

julia> newsite("myBlog", template="basic") #you can choose another name and template

This will create a folder with various directories and elements. It will also activate the environment inside the project. So, if you verify the project with ], it should display the name of your project.

.
├── 404.md            # Page for error 404
├── Manifest.toml     # The typical toml files for Julia development project
├── Project.toml
├── __site            # Generate your full website.
├── _assets           # You can add pictures and images here
├── _css              # All related to styling your website
├── _layout           # All related to the structure of your website
├── _libs             # Here will go all elements for website like katex, searchbar, etc  
├── _rss              # A couple of files related to rss feed, 
├── config.md         # Set Global variables for your website
├── index.md          # Main landing page
├── pages.md          # All your pages / you can create your folder or organize in different way
└── utils.jl          # Julia File for setting some configurations

Finally type:

julia> serve()

It should open your website locally in the browser, and it should look exactly the same as the template website you chose.

Figure 1: starting template

From this point, it's time to delete some files and content. You might also want to add some pages for your projects, about, contact, etc. This is up to you, but for now, we are going to keep just 2 pages: one for the main "about" page and another to host all your posts.

3.2. Cleaning the template

Now, go to the "index.md" page and delete all its content. This page will become your main page, and you can mix HTML and Markdown in this file to add whatever you want to it.

# Welcome to my blog
## I am using Franklin

~~~
    
    
    

~~~


This is an introductory message

You might have noticed that in our main page, there are four links to different pages. You can choose to keep those links or delete them all. However, for the purpose of creating a blog section, let's use one of those links. To do that, follow these steps:

Go to the "header.html" file located in the "layout" folder.
Modify the code in the "header.html" file to something like this:

<header>
<div class="blog-name"><a href="/">a>Amazing Blogdiv>
<nav>
  <ul>
    <li><a href="/">Homea>li>
    <li><a href="/menu1/">Bloga>li>
  ul>
  <img src="/assets/hamburger.svg" id="menu-icon">
nav>
header>

If you're looking to change the background color to something more interesting than white, now is the time to showcase your frontend skills. Follow these steps:

Navigate to the "franklin.css" file.
In the first block of code, add the background color that you prefer. For instance:

:root {
  --block-background: hsl(0, 0%, 94%);
  --output-background: hsl(0, 0%, 98%);
  --small: 14px;
  --normal: 19px;
  --text-color: hsv(0, 0%, 20%);
    background-color: aqua;
}

Finally, after making these modifications, the result should look something like this:

Figure 2: frontend

3.3. Creating your first post

Now, if you're ready to start your own blog, here's how you can set up the "posts" folder to add your articles, create a new folder named "posts" in the same root directory as your other folders. Is important to consider this things.

Inside the "posts" folder, you can add all your articles. You have the flexibility to use both Markdown files and HTML files for your articles.
If you're doing literate programming with tools like Pluto or Jupyter, you can export your notebooks to HTML format and place them in the "posts" folder. This way, anyone can easily view your data science projects.

For now, let's add a file called test1.md inside the posts folder and you can add some text

# This is a title in my first post

So I can write anything

## Here is an introduction

We are going to write some code:

using LinearAlgebra
a = [1, 2, 3, 3, 4, 5, 2, 2]
@show dot(a, a)
println(dot(a, a))

Then, go to the menu1.md file, erase the remaining content, and create a link to the test1.md file. This is as simple as:

If you save it, and navigate to http://localhost:8000/posts/test1/, you should see your post displayed clearly. This page will include your "about" section and the space to write your blog content. Congratulations! You now have a basic understanding of how Franklin works and can make any further edits or modifications you desire.

If you wish to further style your website, please go ahead and customize it to your heart's content.

4. Deployment

Now it's time to host your website in some place. One of the most straightforward options is using GitHub. Here's how you can do it:

Create a Repository: Go to your GitHub account and create an empty repository. When entering the name of your project, you have two paths to choose from:
1. If this is a personal website or organization, the name of your project should be something like username.github.io.
2. You can create your own custom name for your project, like myblog.

If you're unsure which option to choose, I recommend going with option (a) because it's more straightforward. If you choose option (b), you'll need to define a prepath variable in your config.md with the name of that project. For instance: @def prepath = "myblog".

Upload Your Project: Now upload your project to GitHub, following the instructions in your repository.
Configure GitHub Pages: Once you've pushed your project, go to the Settings tab in your repository. Then navigate to GitHub Pages. In the Source dropdown, select gh-pages. If you see a message indicating success, your project is now live.
Check Your Website: You can now open your web browser and enter the link of your project, which would be username.github.io. If you can see your website, congratulations! Your blog is now live on the internet.

By following these steps, you've successfully hosted your Franklin-generated website on GitHub Pages. It's now accessible to anyone with the link, and you can share your content with the world.

4.1. Hosting in a different domain (optional)

If you're hesitant to share your GitHub username due to its lengthy or unconventional extension, or if you prefer a more professional-looking link, you might want to consider an alternative domain, such as .com or .dev. You can purchase a domain and link it to your website. For example, you can use services like Google Domains to find and purchase a domain that suits your preference.

Once you've found and acquired the domain you like, you can proceed to link it to your website. To do this, you need to configure the DNS settings. You can find detailed explanations about custom domains and GitHub Pages in the documentation. In a nutshell, follow these steps:

Go to Google Domains, select your domain, and navigate to the DNS section.
Configure the DNS records, as shown below:

Figure 3: dns_setup

After correctly setting up the DNS records, go to your GitHub project repository's settings, then navigate to Pages and enter your custom domain:

Figure 4: custom_domain

If everything is set up correctly, GitHub will confirm the configuration. In a few minutes, your website should become accessible via your new custom domain.

By following these steps, you'll be able to link a custom domain to your Franklin-generated website, providing a more personalized and professional web presence.

5. RSS and Tags

Now that your website is up and running, setting up an RSS feed is important for people who want to stay updated on your new articles without having to visit your website daily. Tools like Newsboat or Inoreader help users keep track of updates from various websites, making an RSS feed a valuable addition to your blog.

Thankfully, Franklin makes setting up an RSS feed quite simple. All you need to do is go to each page in your "posts" folder and add a small description within +++ brackets, like this:

+++
tags = ["Julia", "Writing"] 

rss_title = "Creating your own blog with Julia and Franklin"
rss_description = "Describing the steps to create your own blog, so you can stop posting your code on Instagram"
rss_pubdate = Date(2023, 8, 10) 
+++

The RSS fields you add will be included in the information extracted by platforms like Newsboat. From these applications, I can read the title, a brief description, and the publication date and all the content if it's available. Additionally, you'll notice a "tags" section. This is also important because it allows users to filter by topics. For example, if you write different blogs about topics ranging from Julia programming to analysis of Shakira's new songs, users can select the topics they're specifically interested in.

To share your blog's RSS feed, you'll need a URL like https://www.yourdomain.com/feed.xml. Make sure to prominently display this URL in your website so that readers can easily find and subscribe to your feed.

5.1. Host your Feed to JuliaBloggers (optional)

Lastly, if you're considering writing about Julia and want to contribute to the community, don't hesitate to share your work. Whether it's a calculator project, a website, a 2D game, or a cutting-edge machine learning algorithm, your contributions will help the Julia community grow and provide valuable insights for others to learn from.

Visit the JuliaBloggers Website and add your information. In the "Feed URL" field, you can use a URL similar to the first example you mentioned, like:

http://indymnv.dev/tag/julia/feed/

Once you've submitted this information, every time you publish a new post on your website, the community will be able to see it. If you want to test this process first, you can use an RSS reader like Newsboat or Inoreader to ensure that your updates are being picked up as expected.

6. Conclusions

I hope you enjoyed reading this article. If you haven't yet created your own website, I hope it serves as motivation to get started, whether you choose to use Franklin or another static site generator. Having your own online space to write about your interests and dive as deep as you like is a rewarding endeavor. Don't hesitate to embark on this journey and create a platform that showcases your passion and expertise. Happy blogging!

7. Acknowledgment

I also want to thank Thibaut Lienart, who is the main developer of Franklin. His work has been incredibly beneficial for the community.

Japonés

Wed, 08 May 2024 00:00:00 +0900

Figure 1: Que buen capitulo wn

En un par de meses más cumpliré 2 años viviendo en Japón, y por supuesto, estar expuesto al idioma y aprendiendolo mientras hay energías. Quiero compartir aquí algunas ideas sobre el idioma.

1. Desafíos

En su momento cuando me dediqué a estudiar inglés para mejorarlo, pensaba que ser bueno en inglés era casi un don, porque el idioma era recomplicado para mi, a pesar de haber tenido una base de estudios en el colegio y en la universidad. Me sentía aavergonzado de mi nivel considerando que compañeros de u y de colegio entendían el idioma, hablaban con extranjeros, consumían contenido en el idioma. Mientras que yo era una bosta de caca sin los subtitulos en español.

Ahora que ya cerré ese capitulo y tengo que estudiar japones, no logro todavía dimensionar que en verdad el inglés era simple y que hay cosas más hardcores como este idioma jajaja. Lo digo, por supuesto, con ciertas introspecciones del idioma. Pero en resumen, el idioma no es necesariamente más dificil, pero tienes que hacer un esfuerzo mayor de memorizar figuras (kanjis) y asimilar una dinámica del lenguaje que es distinta a la que vienes acostumbrado.

El inglés y el español son idiomas que nos guste o no reconocerlos, son muy similares ambas tienen influencias de ramas que distintas (germanica vs romance) pero comparten similitudes como el uso del mismo alfabeto, un grupo importante de palabras cognadas (que basicamente son palabras que tienen el mismo significado y que su escritura y pronunciación son similares, como bank y banco por ejemplo), estructuras gramaticales relativamente similares, etc.

En el japones es bastante diferente, tienes algunos desafíos que enfrentar como:

pocas palabras cognadas, la gran mayoría pueden venir de palabras prestadas del inglés
estructuras gramaticales diferentes, que no tienen muchas veces similitud en como operan con el español y el inglés.
Los niveles de formalidad son más complejos, (mientras que tu te sientes un caballero ulala señor francés, usando "usted"), en el japones el idioma puede sonar muy diferente entre un nivel de formalidad y otro (kenjogo, sonkeigo, etc), incluso cambiando palabras por algunos especificos de la situación.
Y por supuesto los alfabetos (especialmente los kanjis)

Quiero destacar de que si tu ignoraras el sistema de escritura, el japones se vuelve más facil, considerando de que junto con el español la base de la pronunciación es similar, por lo que tendrías una base solida de como suena el idioma, sin embargo, aprender el idioma requiere en mi opinión de que manejes el sistema de escritura y al final es ahí la gran complejidad que yace, esto dado a de que a diferencia del inglés tienes que realizar un importante trabajo previo antes de empezar a consumir todo el contenido que quieras.

2. Mi opinión sobre como aprender el idioma

No voy a declararme una voz autorizada sobre como aprender el idioma, dado de que todavía estoy estudiandolo y por lo tanto todavía no alcanzo el nivel que deseo (que es algo asi como un intermedio avanzado), pero si he hecho algunos progresos como entender los animes del genero slice of life o de escuelas, algunos podcast, también he empezado a entender señales y propagandas que hay en la ciudad, y ya lentamente leer uno que otro texto simple, en lo conversacional creo que puedo defenderme con conversaciones sin ir a algo demasiado específico. Estos son los pasos que recomiendo, no haré mención al tiempo, dado de que realmente depende de tu capacidad de absorver la información y el tiempo que le dedicas a esto:

Aprender hiragana y Katakana, esto es relativamente sencillo, puedes aprenderlo en un fin de semana, y no te preocupas de si puedes o no leerlos 100% de memoria, los veras constantemente así que no te preocupes demasiado por eso masterizarlo de una.
Aprender lo básico de gramática, hay unos libros que son buena base de explicación de como opera el idioma, que son el Minna no Nihongo 1 y 2, date unas semanas o un par de meses para conocer las bases del idioma.
Empieza a consumir anime o series de menores de edad, yo diría cosas que chicos de 2 o 3 años sería sobrio, pero si ya tienes un background sobre el idioma, entonces ver cosas de 10 años como Maruko-chan o Non non Biyori pueden ser buen punto.
Aprende entre 300 a 500 kanjis, esta parte es algunas de las cosas más aburridas puede ser, pero no hay mucho que hacer al respecto, tienes que aprenderte los kanjis de memoria, lo que yo usé por un tiempo fue Wanikani (si no quieres gastar plata puedes usar Anki) que no es nada más que flashcards con un sistema de repetición espaciado, donde puedes estudiar los respectivos significados y lectura de los kanjis, esto además de que será el momento de incorporar vocabulario, que yo recomiendo ir aprendiendo junto con los kanjis, ya que podrás ir aprendiendolos mediante la cohesión de ciertas palabras y será más sencillo, por ejemplo, si aprendes que kanji de enorme (巨 o kyo) y el de persona (人 jin), entonces podrás distinguir una palabra como (巨人 kyojin) que significa literalmente gigante, tal vez recuerdes más eso si viste el anime Shingeki no Kyojin (進撃の巨人).
Empieza a leer mangas, hay algunos que son más para adultos, que ni siquieran usan el furigana (que es la forma en que se leen los kanjis escrito en hiragana o katakana), a mi parecer todavía es demasiado temprano para tirarse a esos mangas, por lo que leer otros mangas más infantiles es mejor para empezar, y por supuesto, evita series de magia, isekai y otro tipo de cosas, le agregan demasiada complejidad a las conversaciones para que alguien que esta recien empezando en el idioma pueda entenderlo.
Sigue iterando hasta alcanzar alrededor de 1000 kanjis masterizados, hacer flashcards sigue siendo importante y te recomiendo seguir paralelamente mientras consumes contenido, aunque en la medida que vayas avanzando irás notando que vas aprendiendo kanjis más raros y que por lo tanto su frecuencia en los textos es mucho menor, en ese momento puede que vayas notando de que el tiempo que le dedicas a aprender esos kanjis va a ser de un retorno menor, por lo que recomiendo que cuando vayas notandolo empieces a pivotear lentamente sobre el consumir contenido y vayas estudiando los kanjis en la medida en que vayan apareciendo.
Avanza estudiando gramática, te recomiendo que vayas de apoco digiriendo los conceptos, puedes usar bunpro por ejemplo u otros libros que corresponden al N3 y N2 del nivel de estudios (no recomendaré ninguno, elige cualquiera), si te aburres, vuelve al consumo de contenido y trata de ir iterando lentamente sobre la gramática.
Empieza a leer una novela ligera, cuando ya tengas una buena cantidad de kanjis bajo el brazo, puedes lanzarte a leer novelas o mini historias que sean puro texto, esto le dará un poco más de dificultad, y tendrás que ponerle más atención a lo que sale en el texto
Lentamente empieza a consumir tanto contenido como sea posible, si eres otaku, entonces esta es tu salsa, aquí yo creo que no hay drama en empezar a consumir el idioma y disfrutarlo, así que de aquí puedes tranquilamente empezar a ser un guatón informático.
A este punto puedes empezar a practicar hablar y escribir.

3. Me recomiendas aprenderlo?

Te digo altiro que salvo que quieres vivir en Japón, o trabajar para una empresa del país, o te guste realmente la cultura nipona y quieras consumirla desde la fuente primaria (sin traducciones), entonces la respuesta es no xd. El japonés es un lenguaje que podría tal vez ayudarte economicamente pero el tiempo que tienes que dedicarle es importante por lo que puede que no valga la pena el esfuerzo (para eso mejor aprende inglés, chino o aleman). Sin embargo, hay algo interesante de la cultura japonesa que merece ser soslayado, y que tiene que ver con la inmensa cantidad de creación de contenido (del bueno eso si) que probablemente solo encontrarás en japones, es decir, el país tiene toda una rica historia sobre temas de filosofía, cultura, ciencias sociales, medicina e ingeniería que solo encontrarás en el idioma y tal vez solamente lo más destacado comercialmente tendrá una traducción al inglés y quizás al español, si te interesa esto, bueno creo que vale la pena el esfuerzo.

Mi configuración personal en Emacs

Mon, 13 Jan 2025 00:00:00 +0900

Figure 1: emacs_meme

Tras unas semanas entendiendo como funciona Emacs, leyendo tutoriales, y descubriendo como otros usuarios usan este software para sus diversas necesidades, he llegado a tener una configuración que me hace sentir relativamente cómodo (obviamente siempre puedes buscar más comodidad al seguir configurandolo, pero todavía estoy aprendiendo).

Aquí desglosaré mi init.el para quién busque inspiración, o simplemente quiera copiar el archivo, el cual tiene alrededor de 250 lineas de código.

1. Generalidades de esta configuración

Si bien no me ataño a ninguna filosofía en particular más de que debe ser cómodo para mi trabajar dentro de Emacs, trato de que: * Solo contenga software que voy a usar: No es necesariamente minimalista, pero tampoco quiero instalar cosas que no necesite, sino, hubiera elegido algo como Doom Emacs. * Mantenerse lo más apegado a los shortcuts originales de Emacs, no he hecho grandes cambios ni agregado muchos nuevos bindings, no pretendo usar Evil mode, dado que todavía me considero un novato con Emacs, por lo que prefiero que se retrate de forma más precisa como Emacs funciona ¹.

2. Dentro del init.el

Para partir lo más básico es lo estético, he decidido remover las barras de herramientas, la barra para scroll y menú, además de algunas configuraciones de fuente y tamaño.

(setq inhibit-startup-message t)

(scroll-bar-mode -1)        ; Disable visible scrollbar
(tool-bar-mode -1)          ; Disable the toolbar
(tooltip-mode -1)           ; Disable tooltips
(set-fringe-mode 10)        ; Give some breathing room

(menu-bar-mode -1)            ; Disable the menu bar

;; Set up the visible bell
(setq visible-bell t)

;; set font decoration
(setq font-lock-maximum-decoration t)

(set-face-attribute 'default nil :font "Iosevka" :height 160)

;; set time
(setq display-time-24hr-format t)
(setq display-time-format "%H:%M")
(display-time-mode 1)

;;set full screen
(set-frame-parameter (selected-frame) 'fullscreen 'maximized)
(add-to-list 'default-frame-alist '(fullscreen . maximized))

Lo siguiente es inicializar el administrador de paquetes y las fuentes donde se descargaran estos paquetes. Aquí tengo seteado melpa (librerías de la comunidad para la comunidad) y elpa (librerías de GNU y adicionales).

;; Initialize package sources
(require 'package)

(setq package-archives '(("melpa" . "https://melpa.org/packages/")
            ( "nongnu" . "https://elpa.nongnu.org/nongnu/")     
                       ("gnu" . "https://elpa.gnu.org/packages/")
             ))

(package-initialize)
(unless package-archive-contents
 (package-refresh-contents))

;; Initialize use-package on non-Linux platforms
(unless (package-installed-p 'use-package)
   (package-install 'use-package))

(require 'use-package)
(setq use-package-always-ensure t)

Una vez seteado las conexiones a melpa y elpa, ya puedes empezar a descargar y usar librerías, soy un ferviente usuario del tema gruvbox, y me gusta usar números relativos para editar ² aunque lo desabilito en algunos unos casos

(use-package doom-themes
     :if window-system
     :ensure t
     :config
     (load-theme 'doom-gruvbox t)
     (doom-themes-org-config)
     (doom-themes-visual-bell-config)
     (menu-bar-mode -1)
     (tool-bar-mode -1)
     (fringe-mode -1)
     (scroll-bar-mode -1))


;; set numbers in the left
(column-number-mode)
(global-display-line-numbers-mode t)
(setq display-line-numbers 'relative)


;; But disable the numbers in some modes
(dolist (mode '(org-mode-hook
               term-mode-hook
               shell-mode-hook
       vterm-mode-hook
               treemacs-mode-hook
       dashboard-mode-hook
               eshell-mode-hook))
 (add-hook mode (lambda () (display-line-numbers-mode 0))))

Aquí agrego un poco de "color" a Emacs al agregar cosas como Ivy para autocompletado, agregar iconos para dired (el administrador de archivos por defecto de Emacs), modificar la barra de estatus, si quieres más color a esta última puedes usar doom-modeline, pero a mi parecer la barra de estatus que viene por defecto es lo suficientemente bueno. ```elisp ;; Set Autocompletion (use-package ivy) (ivy-mode 1)

(setq ivy-use-virtual-buffers t) (setq ivy-count-format "(%d/%d)")

;; basic icons (use-package all-the-icons :ensure t :init)

(use-package all-the-icons-dired :ensure t :init (add-hook 'dired-mode-hook 'all-the-icons-dired-mode))

(use-package all-the-icons-ibuffer :ensure t :init (all-the-icons-ibuffer-mode 1))

;; set delimiters to distinct the language (use-package rainbow-delimiters :hook (prog-mode . rainbow-delimiters-mode))

;; ui to display characters (use-package which-key :defer 0 :diminish which-key-mode :config (which-key-mode) (setq which-key-idle-delay 1))

;; ivy rich (use-package ivy-rich) (ivy-rich-mode 1) ``` Dos librerías fundamentales para incorporar en el workflow de Emacs y que pareciera ser un "must" en la comunidad es Projectile (para administración de proyectos) y Magit (para control de versiones), no los he usado extensivamente pero son buenas piezas de software.

;; Projectile
(use-package projectile
 :ensure t
 :init
 (projectile-mode 1))

;; magit
(use-package magit
 :ensure t
 :config
 (setq magit-push-always-verify nil)
 (setq git-commit-summary-max-length 50)
 :bind
 ("M-g" . magit-status))

La parte que más me interesaba entender de Emacs era Orgmode, así que setié una configuración que me ha funcionado bien, y que las posibilidades son infinitas al momento de customizarlo.

En esta sección me enfoqué en definir una carpeta llamada org, donde he alojado varios archivos para eventos, tareas, etc. y ahí puedo hacer capturas de notas, tareas solo presionando C-c c, también saqué la idea de System Crafters ³ de usar org-refill para poder ir archivando tareas una vez que las completo, también agregué un poco de estilo a los archivos .org (para más placer).

;; org mode
(global-set-key (kbd "C-c l") #'org-store-link)
(global-set-key (kbd "C-c a") #'org-agenda)
(global-set-key (kbd "C-c c") #'org-capture)

(setq org-ellipsis " ")
(setq org-src-fontify-natively t)
(setq org-src-tab-acts-natively t)
(setq org-confirm-babel-evaluate nil)
(setq org-export-with-smart-quotes t)
(setq org-src-window-setup 'current-window)
(setq org-directory "~/org")
(setq org-agenda-files '("events.org" "tasks.org" "birthday.org"))
(setq org-agenda-start-with-log-mode t) ;; set log for completion in agenda
(setq org-log-done 'time)
(setq org-log-into-drawer t)

;; After complete a task send to Archive or other file listed
(setq org-refile-targets
     '(("archive.org" :maxlevel . 1)
   ("tasks.org" :maxlevel . 1)))
;; after send the task completed, save all the org files
(advice-add 'org-refile :after 'org-save-all-org-buffers)

;; Set templates for fast note taking and other
(setq org-capture-templates
     '(("t" "Todo" entry (file+headline "~/org/tasks.org" "INBOX")
        "* TODO %?\n  %i\n  %a")
       ("j" "Journal" entry (file+datetree "~/org/journal.org")
        "* %?\nEntered on %U\n  %i\n  %a")))


(add-hook 'org-mode-hook 'org-indent-mode)


;; sintactic sugar for bullets
(require 'org-bullets)
(add-hook 'org-mode-hook (lambda () (org-bullets-mode 1)))

Uso de Dashboard a la hora de abrir Emacs para poder ver archivos, proyectos y tareas pendientes, además de una frase bíblica para más inspiración.

Figure 2: dashboard

;; Dashboard at open emacs
(use-package dashboard
 :ensure t
 :config
   (dashboard-setup-startup-hook)
   (setq dashboard-center-content t)
   (setq dashboard-startup-banner 'official)
   (setq dashboard-items '((recents  . 5)
               (projects . 5)
               (agenda    . 5)))
   (setq dashboard-banner-logo-title "Do you see a man skillful in his work? He will stand before kings; he will not stand before obscure men. (Proverbs 22:29)"))

Permitir procesos asincronos elisp ;; allow async process (use-package async :ensure t :init (dired-async-mode 1))

Syntax highlighting para markdown elisp ;; Use markdown (use-package markdown-mode :ensure t :mode ("README\\.md\\'" . gfm-mode) :init (setq markdown-command "multimarkdown"))

Para leer archivos pdf, Emacs tiene incorporado un ereader, con funcionalidades básicas, sé que existe una librería con mayor funcionalidades llamado pdf-tools, pero no he tenido la necesidad de usarlo ⁴, sin embargo, porque tengo varios epubs que leo, instalé nov.el para dicho propósito.

También he incorporado emms para escuchar música, hace el trabajo de base, pero todavía no me he tomado el tiempo de revisar a fondo como funciona.

Y finalmente agregué elfeed para leer rss de distintas noticias y blogs, definitivamente me ha gustado más que newsboat. Dado que tiendo a leer blogs de estadística y análisis de datos que por su naturaleza contienen muchas imagenes, no requiero de abrir el navegador para poder tener un detalle completo sobre este.

Figure 3: utilities

;; Nov.el to read Epub
(use-package nov
 :ensure t)
(add-to-list 'auto-mode-alist '("\\.epub\\'" . nov-mode))

;; To display music as a gigachad
(use-package emms)
(emms-all)
(setq emms-player-list '(emms-player-mpv)
     emms-info-functions '(emms-info-native))

;; To read RSS
;; Somewhere in your .emacs file
(use-package elfeed)
(setq elfeed-feeds
     '(
   ("https://planet.emacslife.com/atom.xml" Emacs Blog)
   ("https://news.ycombinator.com/rss" News)
   ))
;; set global keybinding for elfeed
(global-set-key (kbd "C-x w") 'elfeed)

Ahora para algunas de las partes más relacionadas a desarrollo y análisis de datos, he incorporado ess (Emacs speaks Statistic), librería fundamental para trabajar con R y otros software ⁵. Para Python he incorporado elpy que es una librería que viene con lo necesario para desarrollo en Python, también sé de la existencia de ein para Jupyter Notebooks, no lo he probado todavía, tal vez lo haga en algún futuro. Para Julia hay varias discusiones dando vueltas sobre como aprovechar Emacs ⁶, por ahora tengo solo seteado 2 librerías, julia-mode y julia-snail, el primero es solo syntax highlighting y el segundo permite interactuar con el REPL, de una forma similar a la que usarías Slime, haciendo evaluaciones linea por linea cuando sea requerido.

;; statistical mode in emacs
(use-package ess)
(use-package company
 :config
 (add-hook 'after-init-hook 'global-company-mode))

;; python development
(use-package elpy
 :ensure t
 :init
 (elpy-enable))

;; Julia mode
(use-package julia-mode
 :ensure t)

(use-package julia-snail
 :ensure t
 :hook (julia-mode . julia-snail-mode))
(setq-default julia-snail-executable "/home/navi/.juliaup/bin/julia")

Emacs por alguna extraña razón tiene incorporado 3 terminales, eshell, shell, term. Sé que existen usos específicos para cada uno aunque aunque no estoy del todo claro ⁷, especialmente por ese extraño patrón de que puedes borrar lineas de código escritas en la terminal, o porque tienes que cambiar tus bindings al momento de moverte de un buffer a otro (C-x a C-c), o no ser capaz de abrir vim dentro estas terminales ⁸, sin embargo, leí comentarios positivos con respecto a Vterm, y decidí instalarlo y probarlo, hasta ahora es el que se ha ajustado mejor a mis preferencias. Y por lo tanto, he usado por defecto ⁹

;;A new  terminal with more capabilities
(use-package vterm
  :ensure t)

Finalmente, una cosa que a menudo me encuentro haciendo es copiando y pegando lineas completas de texto, tomé esta función customizada de BugsWriter para poder alcanzar ese propósito.

;; custom function to copy whole line with C-c w l
 (defun copy-whole-line ()
   (interactive)
   (save-excursion
     (kill-new
      (buffer-substring
    (point-at-bol)
    (point-at-eol)))))
(global-set-key (kbd "C-c w l") 'copy-whole-line)

Notar algo que apoyo completamente y doy como consejo es reemplazar el Ctrl por la tecla CapsLock, realmente resolverá un montón de dolores que tienes con Emacs y su extraña posición con las teclas lideres. Así que por favor antes que te lances a usar emacs por primera vez, te recomiendo hacer este cambio.

3. Trabajos y exploraciones futuras

Algunas cosas que quiero seguir explorando y probablemente incorporando en Emacs en caso de que se ajusten bien:

Operar email dentro de Emacs
Org Roam, bastantes cosas positivas he escuchado sobre este software para tomar notas que parece inspirarse en el Zettelkasten method. También Denote es una opción.
Eglot o cualquier servidor del lenguaje que funcione.
Org Babel, para literate programming.
Integraciones con Quarto.
EXWM, un windows manager dentro de Emacs, parece innecesario en mi caso de estar usando Sway, pero ¿Por qué no probarlo?.
Aprender Elisp, hasta ahora he usado principalmente librerías que la comunidad o GNU han creado, pero el verdadero valor está en que puedas aprender Elisp y hackear por tu cuenta.
Incorporar algún LLM ¹⁰.

4. Conclusiones

Dentro de todo, Emacs ha probado ser un interesante reemplazo al flujo de trabajo que estaba usando con Vim + Tmux + amigos, muchas de las cosas que me tomaron tiempo aprender a usar con este workflow, en Emacs fue resuelto casi en un par de semanas, por supuesto, hay un sesgo acá de aprendizaje de herramientas previas que sin dichas experiencias hubiera probablemente ralentizado mi aprendizaje de Emacs.

Otra cosa que cabe notar, es que si vienes con una mente abierta, Emacs puede ser realmente divertido de usar, Elisp realmente es un buen lenguaje para este software y la mística de "hackear" tu editor es algo que de verdad puede llevarte a inspirar hacer cosas por tu cuenta y darte absoluta libertad, convertir un editor en tu workshop digital tiene sus dificultades, pero estoy seguro de que vale la pena si te atreves a ensuciarte las manos.

Footnotes:

https://www.masteringemacs.org/article/beginners-guide-to-emacs

Practica que vengo usando desde vim

Buen canal para aprender sobre Emacs, Guix, Lisp, entre otros. https://systemcrafters.net/

⁴

Tampoco soy mucho de andar destacando y tomando notas en el pdf, creo que es mejor tomar notas por separado.

⁵

Pero parecieran no estar tan depurados como R

⁶

https://discourse.julialang.org/t/emacs-and-julia-how-to/123623

⁷

Aquí hay un artículo que lo explica sencillamente http://xahlee.info/emacs/emacs/emacs_shell_vs_term_vs_ansi-term_vs_eshell.html

⁸

Si, a veces quiero hacer algo como eso.

⁹

Aunque surfeando por la web, me encontré con este artículo que me hace replantear la idea de profundizar con Eshell: https://howardism.org/Technical/Emacs/eshell-why.html

¹⁰

Preferiblemente si es que puedo correrlo localmente.

Arch, Debian, Suckless and wrapping up the Linux experience

Fri, 19 Apr 2024 00:00:00 +0900

Figure 1: tony

If you want to read the spanish version visit the following link

After spending some time tinkering with a second-hand ThinkPad T480 that I ended up buying in a remote corner of the city where I live, I've been trying out different Linux distributions and desktop environments, but overall I'll focus on two distributions that in my opinion are the most important: Debian and Arch. In addition to this, I wanted to try out a new Tiling Window Manager that had also caught my attention, as well as a set of software that embodies an interesting philosophy: Suckless.

1. Some experiences with Arch and Debian

To start with the distributions, Debian is incredibly simple to get started with, the installation is approachable, and I must admit that I like that it is (or pretends to be) 100% free software. Furthermore, once you go through the installation process, the desktop you choose just works, no surprises, it's rock solid, and of course, you don't expect much on the horizon regarding updates. In fact, every time I did an apt update, everything was simply up to date. So it's a rather boring distribution if you want to be constantly surprised with new things, but for me, this is quite positive, as it maintains certainty in the system, you know it won't fail unless you do something stupid. Regarding the software you install, obviously this is where you notice the gap, since Debian is quite conservative when it comes to bringing software into its main repository and therefore it's not uncommon to have software versions that may be 10 or 12 months behind the current development version. This may not be a big problem, but each software has a different development speed, and in a year you can have significant differences. Also, if you've been carrying more updated versions of programs that have some kind of config file, that can be a problem. An example where I noticed this was when I wanted to install Alacritty and port my .toml file with my configurations, when using it, I realized that the changes weren't applied. The reason was that the version of Alacritty that comes from Debian's apt was an old version that used .yml files, so I had to 1) adapt my configuration files to this version, which for this case wasn't complicated, or 2) compile from source.

Compiling from source isn't a bad approach, however, it means that you're at the mercy of your package manager taking care of future updates, and obviously that can represent a risk. An important issue with a system like Linux is that you have your software updated and operating harmoniously with your operating system. Of course, there's nothing wrong with compiling from source, I've done it with some software, but it's important to note that it's not the most desirable. However, a positive aspect to recognize is that Debian has a vast repository, which reduces the chances of having to resort to installing from source code.

On the other hand, I'm now using Arch (btw) so I'm still in a process of experimentation with this variant. Arch is different in its installation process, although now there's a script that simplifies the process, the manual installation is in the terminal by typing commands and where you have to follow the instructions from the wiki, it's not that difficult, but obviously it's more likely that you can make a mistake along the way and therefore mess up, in fact, I messed up a couple of times because I didn't configure the Nvidia driver properly and therefore my system failed, so I had to start the process again.

If you want up-to-date software, this is the distribution and even if you want to access proprietary software, Arch has no problems, it's a distribution agnostic and therefore has a laissez-faire mentality towards users, I can run the analog of apt update which in this case is Pacman -Syu every day and there may be new updates. Obviously with software like Alacritty or Neovim I didn't have to resort to compiling it from source because Arch itself provides recent updates which is also a pleasure not to have to directly intervene with the code, another thing is that in addition to the base repository, Arch has the AUR which is another repository where the community can upload software and of course that allows your possibilities to expand, however in my experience, it's in the AUR where you have more chances of software failure.

Here are some other examples of software that allowed me to compare distributions. As I like to watch anime, I wanted to install ani-cli which is an application to watch content from the terminal (you have to thank that the hacker community is quite otaku), unfortunately, this software in Debian is not considered in its main repository yet and will apparently be added in version 13, so one option was to compile it, but I finally gave up on it. On the other hand, with Arch there was no problem, the repo was in AUR so the installation was quite simple. The second case was Anki, a flashcard application, which I like and have used a lot to learn languages so I thought about installing it on my pc as well (although the Android app is quite good to be honest), unfortunately Anki is a mess, I went to the documentation and tried to install it following the instructions but had problems in the process, fortunately Debian has Anki in its main repository, so installing it with apt was the best option, on the other hand Arch had it in the AUR, however, I had problems with the installation so I didn't continue with it. The third and last case was Librewolf, in this case Debian had no problems, by following the steps in the documentation the issue worked without problems, Arch had it in the AUR, and the installation process took quite some time and finally I had problems with that but the binary worked fine.

As a side note, I must praise the Arch Linux Wiki, you may or may not like Arch, but I think no one can deny that its documentation is of a high level, it helped me to see a specific case where I wanted to see how to install the Japanese keyboard on it with the kanas specific to the writing of this language, I checked the documentation in Debian and Arch, finally the latter gave me clear instructions on how to configure it correctly, besides that on other occasions when I've had problems with something in Debian/Ubuntu/Fedora or even Arch itself the Wiki has been helpful.

Finally, I want to highlight that both have the possibility of minimal installations, so if you have a laptop with hardware restrictions, I think there's no problem with any of these variants for you to do a minimal installation and instead of using a bloated desktop environment, you can use a window manager that will be much lighter and more efficient in resources like Sway, i3, or dwm. Although I wrote about Sway in another post, now I want to focus on dwm and another set of software that belongs to suckless.

2. Suckless: Software that suckless

Suckless is a community of hackers who have built a set of software based on the concepts of minimalism, simplicity, and frugality, part of this decision you can read in their philosophy, but to try to get to the point, all the code is written in C, and there are no configuration files, if you want to install this, you download the code and compile it, if you want to make modifications to something, you go straight to the C source code and modify it and recompile it. The most important software (in my opinion) are:

st a terminal emulator
dwm the window manager
dmenu the dynamic menu that you can use to run your programs

Something you have to notice is that this software has minimal features, for example, St doesn't have a way to scroll, nor does it have a way to make your terminal transparent, if you want to achieve this, you have two options: the first is to write the code yourself, but if you're like me and have no idea about C then accessing the patches that the community has created and improved over time is the option, so for example, to the terminal I can add transparency with alpha and a gruvbox color scheme. To dwm I can add patches to add gaps with useless gaps. all this just with the command

patch -p1 < path/to/patch

It should be noted that starting to add patches can lead to errors, so you will have to be the one to make the corresponding adjustments for the failures, and in some cases if you have several patches in mind, you have to consider that there is a more appropriate order to do it to reduce the potential number of errors. Obviously the idea is not that you go and take a collection of patches that you "like" and apply them, but that you be much more austere and select them according to a real need that arises, for example, I only applied useless gaps to dwm for a bit of aesthetics, but the truth is that dwm by itself has been fantastic, I haven't had the intention of adding another patch for now, I haven't needed to add scrolling to st, since I use tmux quite a bit, which serves that function as a substitute. So if you like to add things to your application just because, it's possible but you'll have to deal with debugging the errors that occur while adding the patches.

Other software that I have been adding and that follows a similar philosophy:

nsxiv: To display images
scrot: screenshots
feh: set an image as background
cmus: music player
ly: minimal session manager.
mpv: Media player
slock: lock your system with a password.
xclip: X11 clipboard

Among others…

Obviously I have to point out that these exercises for a machine that isn't really that old and has relatively decent hardware maybe is considered exaggereted, as an exercise it was worth it and it taught me a lot about minimal installations and how you can reach customization levels while keeping resource usage to a minimum and in fact I will continue using this configuration until I get bored, but if you're not interested in that Gnome or KDE run quite well on a machine like this.

3. Conclusions at the end

Regarding Arch and Debian I don't have a clear preference yet, both have their pros and cons, for now I'll continue using Arch to continue evaluating how it goes. Although there are other distributions, even some more hardcore ones like Gentoo, for now my curiosity hasn't reached that point yet.

Suckless software is an interesting perspective, quite radical to market trends and for a niche taste, I can't imagine many people wanting to get into this point, but for those who do, I'm sure it's a good exercise to use something like dwm, inspect the C code, customize, add patches, etc. If you can do that, the idea of making your own software for a specific personal need may not be so far-fetched.

I've also had a lot of fun with Linux over the past 4 months, it has taught me a lot about how an operating system works especially when I've wanted to go to slightly deeper levels and start eliminating the use of interfaces, I'm not going to call myself a "power user", but certainly t here's some knowledge that I value and I understand a bit more about those people who want to know more about their system and adjust it to their liking and have absolute control over it. If there's anyone interested in knowing what my "learning path" was, which by the way was not planned at all it was just pure curiosity, I would describe it as follows:

Install free and open source software on your own machine, even if you're on MacOS or Windows. Install and familiarize yourself with their basic uses and if you can include it in something you do daily even better. For example, Vim, Python, R, LibreOffice, etc.
Learn basic commands in a terminal: commands like cd, mv, rm, cp, ls. I use them constantly nowadays and I would recommend reading a bit about them and doing some tests to remove the fear of the black screen, then you can add new commands like curl, grep, sed, etc.
Install Ubuntu or another friendly distribution: I used Ubuntu at the time, there are people who criticize it currently for various reasons. But something that Ubuntu has is that its installation process was super friendly, a modern look, there's plenty of documentation on the internet and it works well. Linux Mint seems like another good alternative to Ubuntu, but I don't have experience with that distribution. This is for a start, later if you want to move to another distribution nothing will prevent you from doing it
Get familiar with the package manager: The first way to install software is from the terminal using your package manager, if you're on Ubuntu or Mint you'll have to use something like sudo apt update and then sudo apt install yoursoftware, there are other alternatives to install software but this should be your priority method.
Install software by compiling source code: Take a software that interests you, go to its documentation or the repository where it is and read about its installation process, probably besides the package manager it will have a space where it will give instructions on how to do it, follow those instructions.
Use a Tiling Window Manager: There are two major desktop environments in Linux GNOME and KDE, both have a lot of features and you can easily navigate in them, adjust brightness, connect to wifi, explore your folders with files, screenshots, etc. With a twm you won't have any of that, and you'll have to look up how to execute each of these commands, you need to connect your mouse via bluetooth, one possible solution is to install bluez and bluez utils, open the terminal and use the bluetoothctl command, there are also GUI options, but it will depend on you to find a way to execute it. Play with this and if you want try to do a ricing (configure your environment to look aesthetically attractive).
Create a remote server and run some programs on them: If you have an old laptop, or a raspberry pi, or rent a vps, you can try to create a remote server where you can host some programs, this is the path to another world that is self-hosting, I'm quite a newbie in this, but there are a couple of experiences that can teach you something, for example create a personal website and host it on your server, buy your domain and redirect it to your website. One thing I did for example was to get rid of Kindle and host my books in Calibre, so now I can access my personal library from my own server. With this you can learn a bit about connecting via ssh, nginx, dns, docker, etc. There are people who host a bunch of applications from a youtube frontend, emails, vpn, etc. There you can see for yourself what to do.
Install Arch Linux manually: If you installed Ubuntu before, you'll notice that the process is quite simple in general, installing Arch is more complex, but it will teach you how to make an operating system from scratch, so you can understand what some of the base libraries are for, make the partitions manually, configure network, timezone, keyboard, decide between X11 or Wayland, etc. Try to use it for a period of time as your main machine and use it for everything you can.
Contribute to Linux: I plead guilty of not doing it yet, I suppose this is a note for the future…

Anyway, it's been an interesting journey and I think I'll continue using Linux for my personal machines. I'll probably continue to see some self-hosting things, but I think I've had enough with ricing and trying distributions, it's time to move on to another topic.

World Happiness Report - EDA and clustering with Julia

Wed, 23 Nov 2022 00:00:00 +0900

The purpose of this post is to show Julia as a language for data analysis and Machine Learning. Sadly Kaggle does not support Julia Kernels (hopefully, they will add it in the future). Therefore I wanted to take advantage of this space to show a reimplementation of Python/R Notebooks to Julia. In this context, I took data on happiness in countries in 2021 and some factors considered in this exciting survey.

You can get the dataset in Kaggle
The full code is in my Github

1. Packages used

I'm using Julia version 1.8.0 in this project, and the library versions are in the Project.toml, there are some installed that I didn't end up using for this analysis, but these are the important ones

using DataFrames
using DataFramesMeta
using CSV
using Plots
using StatsPlots
using Statistics
using HypothesisTests
Plots.theme(:ggplot2)

Let's start reading the file.

df_2021 = DataFrame(CSV.File("./data/2021.csv", normalizenames=true))

You can see the dataset in the REPL.

julia> df_2021 = DataFrame(CSV.File("./data/2021.csv", normalizenames=true))
149×20 DataFrame
 Row │ Country_name    Regional_indicator            Ladder_score  Standard_error_of_ladder_score  upperwhi ⋯
     │ String31        String                        Float64       Float64                         Float64  ⋯
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ Finland         Western Europe                       7.842                           0.032         7 ⋯
   2 │ Denmark         Western Europe                       7.62                            0.035         7
   3 │ Switzerland     Western Europe                       7.571                           0.036         7
   4 │ Iceland         Western Europe                       7.554                           0.059         7
   5 │ Netherlands     Western Europe                       7.464                           0.027         7 ⋯
   6 │ Norway          Western Europe                       7.392                           0.035         7
   7 │ Sweden          Western Europe                       7.363                           0.036         7
   8 │ Luxembourg      Western Europe                       7.324                           0.037         7
   9 │ New Zealand     North America and ANZ                7.277                           0.04          7 ⋯
  10 │ Austria         Western Europe                       7.268                           0.036         7
  11 │ Australia       North America and ANZ                7.183                           0.041         7
  12 │ Israel          Middle East and North Africa         7.157                           0.034         7
  13 │ Germany         Western Europe                       7.155                           0.04          7 ⋯
  14 │ Canada          North America and ANZ                7.103                           0.042         7
  ⋮  │       ⋮                      ⋮                     ⋮                      ⋮                      ⋮   ⋱
 136 │ Togo            Sub-Saharan Africa                   4.107                           0.077         4
 137 │ Zambia          Sub-Saharan Africa                   4.073                           0.069         4
 138 │ Sierra Leone    Sub-Saharan Africa                   3.849                           0.077         4 ⋯
 139 │ India           South Asia                           3.819                           0.026         3
 140 │ Burundi         Sub-Saharan Africa                   3.775                           0.107         3
 141 │ Yemen           Middle East and North Africa         3.658                           0.07          3
 142 │ Tanzania        Sub-Saharan Africa                   3.623                           0.071         3 ⋯
 143 │ Haiti           Latin America and Caribbean          3.615                           0.173         3
 144 │ Malawi          Sub-Saharan Africa                   3.6                             0.092         3
 145 │ Lesotho         Sub-Saharan Africa                   3.512                           0.12          3
 146 │ Botswana        Sub-Saharan Africa                   3.467                           0.074         3 ⋯
 147 │ Rwanda          Sub-Saharan Africa                   3.415                           0.068         3
 148 │ Zimbabwe        Sub-Saharan Africa                   3.145                           0.058         3
 149 │ Afghanistan     South Asia                           2.523                           0.038         2

To see the columns name, simply use

names(df_2021)

getting a vector with all column names

julia> names(df_2021)
20-element Vector{String}:
 "Country_name"
 "Regional_indicator"
 "Ladder_score"
 "Standard_error_of_ladder_score"
 "upperwhisker"
 "lowerwhisker"
 "Logged_GDP_per_capita"
 "Social_support"
 "Healthy_life_expectancy"
 "Freedom_to_make_life_choices"
 "Generosity"
 "Perceptions_of_corruption"
 "Ladder_score_in_Dystopia"
 "Explained_by_Log_GDP_per_capita"
 "Explained_by_Social_support"
 "Explained_by_Healthy_life_expectancy"
 "Explained_by_Freedom_to_make_life_choices"
 "Explained_by_Generosity"
 "Explained_by_Perceptions_of_corruption"
 "Dystopia_residual"

The features of this dataset are as follow:

Country_name: Name of the country
Regional_indicator: The region to which the country belongs.
Ladder_score: The English wording of the question is "Please imagine a ladder, with steps numbered from 0 at the bottom to 10 at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. On which step of the ladder would you say you personally feel you stand at this time?", this metric represent the average of this response by country
Standard_error_of_ladder_score: This metric represent the standard error of the Ladder_score metric.
upperwhisker: Refers to the upper part of a box plot of the Ladder_score metric
lowerwhisker Refers to the lower part of a box plot of the Ladder_score metric
Logged_GDP_per_capita: GDP per Capita Registered to the date
Social_support: Average of the question based on: 'If you were in trouble, do you have relatives or friends you can count on to help you whenever you need them, or not?' The response is 0 for 'no' and 1 for 'yes'.
Healthy_life_expectancy: Average lifespan by country, information extracted from the World Health Organization's (WHO) Global Health Observatory data repository.
Freedom_to_make_life_choices: National average of responses to the GWP question "Are you satisfied or dissatisfied with your freedom to choose what you do with your life?"
Generosity: Is the residual of regressing national average of response to the GWP question "Have you donated money to a charity in the past month?" on GDP per capita.
Perceptions_of_corruption: "Is corruption widespread throughout the government or not" and "Is corruption widespread within businesses or not?" The overall perception is just the average of the two 0-or-1 responses.
The variable 'Dystopia' and the explained variables that come from a built-in regression model are not taken into consideration for this project.

To see what is a regional indicator, we can see how every country is grouped.

julia> unique(df_2021.Regional_indicator)
10-element Vector{String}:
 "Western Europe"
 "North America and ANZ"
 "Middle East and North Africa"
 "Latin America and Caribbean"
 "Central and Eastern Europe"
 "East Asia"
 "Southeast Asia"
 "Commonwealth of Independent States"
 "Sub-Saharan Africa"
 "South Asia"

Let's do a simple operation with the dataframe getting the number of countries by regional indicator and sorting those

sort(
    combine(groupby(df_2021, :Regional_indicator), nrow), 
    :nrow
)

Getting this output

julia> sort(
           combine(groupby(df_2021, :Regional_indicator), nrow),
           :nrow
       )
10×2 DataFrame
 Row │ Regional_indicator                 nrow
     │ String                             Int64
─────┼──────────────────────────────────────────
   1 │ North America and ANZ                  4
   2 │ East Asia                              6
   3 │ South Asia                             7
   4 │ Southeast Asia                         9
   5 │ Commonwealth of Independent Stat…     12
   6 │ Middle East and North Africa          17
   7 │ Central and Eastern Europe            17
   8 │ Latin America and Caribbean           20
   9 │ Western Europe                        21
  10 │ Sub-Saharan Africa                    36

With this, we can see a more significant number of countries in Sub-Saharan Africa and only a smaller group of countries in North America and ANZ.

Now, let's try to slice our data. We will create a data frame called float_df that contains only the Float64 variables but excludes the "explained_" variables. This new dataframe will help us with some operations later.

#Get all columns Float64
float_df = select(df_2021, findall(col -> eltype(col) <: Float64, eachcol(df_2021)))

#Take away the Explained variables
float_df = float_df[:,Not(names(select(float_df, r"Explained")))]

Let's make our first plot.

scatter(
    df_2021.Social_support,
    df_2021.Ladder_score,
    size = (1000,800),
    label="country",
    xaxis = "Social Support",
    yaxis = "Ladder Score",
    title = "Relation between Social Support and Happiness Index Score by country"
)

Figure 1: scatterplot with ladder score and social support

If we want a view of all float variables in several histograms, we can add this code using Statsplots.

N = ncol(float_df)
numerical_cols = Symbol.(names(float_df,Real))
@df float_df Plots.histogram(cols();
                             layout=N,
                             size=(1400,800),
                             title=permutedims(numerical_cols),
                             label = false)

Figure 2: Histogram of all variables

And If we want to compare it with boxplots.

@df float_df boxplot(cols(), 
                     fillalpha=0.75, 
                     linewidth=2,
                     title = "Comparing distribution for all variables in dataset",
                     legend = :topleft)

Figure 3: Boxplot all variables

Without going into so much detail, we can affirm that the Ladder Score is the variable related to the result of the survey on the degree of happiness in the country (our dependent variable). Explained variables correspond to the preprocessing to build the Ladder Score, for this reason, we remove them from the dataframe and will hold with only the raw data.

What are the top 5 countries and bottom 5?

# Top 5 and bottom 5 countries by ladder score
sort!(df_2021, :Ladder_score, rev=true)
plot(
    bar(
        first(df_2021.Country_name, 5 ),
        first(df_2021.Ladder_score, 5 ),
        color= "green",
        title = "Top 5 countries by Happiness score",
        legend = false,
    ),
    bar(
        last(df_2021.Country_name, 5 ),
        last(df_2021.Ladder_score, 5 ),
        color ="red",
        title = "Bottom 5 countries by Happiness score",
        legend = false,
    ),
size=(1000,800),
yaxis = "Happines Score",
)

Figure 4: top5 and bottom 5

And the classic heatmap for correlation with the following function.

function heatmap_cor(df)
    cm = cor(Matrix(df))
    cols = Symbol.(names(df))

    (n,m) = size(cm)
    display(
    heatmap(cm, 
        fc = cgrad([:white,:dodgerblue4]),
        xticks = (1:m,cols),
        xrot= 90,
        size= (800, 800),
        yticks = (1:m,cols),
        yflip=true))
    display(
    annotate!([(j, i, text(round(cm[i,j],digits=3),
                       8,"Computer Modern",:black))
           for i in 1:n for j in 1:m])
    )
end

Figure 5: heatmap

And now, we can build a function where we can get the mean ladder score by regional indicator and compare it with the distribution of all countries.

function distribution_plot(df)
    display(
        @df df density(:Ladder_score,
        legend = :topleft, size=(1000,800) , 
        fill=(0, .3,:yellow),
        label="Distribution" ,
        xaxis="Happiness Index Score", 
        yaxis ="Density", 
        title ="Comparison Happiness Index Score by Region 2021") 
    )
    display(
        plot!([mean(df_2021.Ladder_score)],
        seriestype="vline",
        line = (:dash), 
        lw = 3,
        label="Mean")
    )
    for element in unique(df_2021.Regional_indicator)
        display(
            plot!(
            [mean(mean([filter(row->row["Regional_indicator"]==element, df).Ladder_score]))],
            seriestype="vline",
            lw = 3,
            label="$element") 
        )
    end
end

Figure 6: distribution region

Suppose we want to try the same idea but with countries. In that case, we can take advantage of multiple dispatch and create a function that receives a list of countries and creates a variation of the distribution with countries.

function distribution_plot(df, var_filter, list_elements)
    display(
        @df df density(:Ladder_score,
        legend = :topleft, size=(1000,800) , 
        fill=(0, .3,:yellow),
        label="Distribution" ,
        xaxis="Happiness Index Score", 
        yaxis ="Density", 
        title ="Happiness index score compare by countries 2021") 
    )
    display(
        plot!([mean(df_2021.Ladder_score)],
        seriestype="vline",
        line = (:dash), 
        lw = 3,
        label="Mean")
    )
    for element in list_elements
        display(
            plot!(
            mean([filter(row->row[var_filter]==element, df).Ladder_score]),
            seriestype="vline",
            lw = 3,
            label="$element") 
        )
    end
end

Let's test our new function, comparing three countries.

distribution_plot(df_2021, "Country_name", ["Chile",
                                            "United States",
                                            "Japan",
                                           ])

Figure 7: distribution countries

Here we can see how the USA has the highest score, followed by Chile and Japan.

To end the first part, let's apply some statistical tests. We will use an equal variance T-test to compare distribution from different regions. The function is as follows.

# Perform a simple test to compare distributions
# This function performs a two-sample t-test of the null hypothesis that s1 and s2 
# come from distributions with equal means and variances 
# against the alternative hypothesis that the distributions have different means 
# but equal variances.
function t_test_sample(df, var, x , y)
    x = filter(row ->row[var] == x, df).Ladder_score
    y = filter(row ->row[var] == y, df).Ladder_score
    EqualVarianceTTest(vec(x), vec(y))
end

We will have this output if we compare Western Europe and North America and ANZ.

t_test_sample(df_2021, "Regional_indicator", "Western Europe", "North America and ANZ")

julia> t_test_sample(df_2021, "Regional_indicator", "Western Europe", "North America and ANZ")
Two sample t-test (equal variance)
----------------------------------
Population details:
    parameter of interest:   Mean difference
    value under h_0:         0
    point estimate:          -0.213595
    95% confidence interval: (-0.9068, 0.4796)

Test summary:
    outcome with 95% confidence: fail to reject h_0
    two-sided p-value:           0.5301

Details:
    number of observations:   [21,4]
    t-statistic:              -0.6374218416101513
    degrees of freedom:       23
    empirical standard error: 0.3350924366753546

We don't have enough evidence to reject the hypothesis that these samples come from distributions with equal means and variance. On another side, if we try comparing Western Europe with South Asia, we can see this:

julia> t_test_sample(df_2021, "Regional_indicator", "South Asia", "Western Europe")
Two sample t-test (equal variance)
----------------------------------
Population details:
    parameter of interest:   Mean difference
    value under h_0:         0
    point estimate:          -2.47305
    95% confidence interval: (-3.144, -1.802)

Test summary:
    outcome with 95% confidence: reject h_0
    two-sided p-value:           <1e-07

Details:
    number of observations:   [7,21]
    t-statistic:              -7.576776118465833
    degrees of freedom:       26
    empirical standard error: 0.32639840222022687

In this case, we can reject that hypothesis.

2. Clustering

Now we will cluster the countries using the popular algorithm Kmeans. My first option was to use clustering.jl. However, determining the ideal number of clusters is necessary to get the Wcss (within-cluster sum of the square). With this, we can evaluate it with the elbow method, so I used Scikit-learn wrapper. I also include an issue. Well, let's continue with the last part. I started adding some libraries.

using Random
using ScikitLearn
using PyCall

@sk_import preprocessing: StandardScaler
@sk_import cluster: KMeans

Let's take out from the float_df all the variables related to Ladder_score, and keep only the variables considered in the survey.

select!(float_df, Not([:Standard_error_of_ladder_score, 
                           :Ladder_score, 
                           :Ladder_score_in_Dystopia, 
                           :Dystopia_residual]))

To train our model, we need to standardize the data, and then we will create a list to retrieve the wcss in every iteration. The function is as follows:

function kmeans_train(df)
    X = fit_transform!(StandardScaler(), Matrix(df))

    wcss = []
    for n in 1:10

        Random.seed!(123)
        cluster =KMeans(n_clusters=n,
                        init = "k-means++",
                        max_iter = 20,
                        n_init = 10,
                        random_state = 0)
        cluster.fit(X)
        push!(wcss, cluster.inertia_)
    end
    return wcss
end

Let's invoke the function and plot the wcss.

wcss = kmeans_train(float_df)

plot(wcss, title = "wcss in each cluster",
    xaxis = "cluster",
   yaxis = "Wcss")

Figure 8: Elbow Method

In this case, I decided to go for three clusters. We can abuse make use of multiple dispatch again, adding n for a defined number of clusters.

function kmeans_train(df, n)
    X = fit_transform!(StandardScaler(), Matrix(df))

    Random.seed!(123)
    cluster =KMeans(n_clusters=n,
                    init = "k-means++",
                    max_iter = 20,
                    n_init = 10,
                    random_state = 0)
    cluster.fit(X)
    return cluster
end

cluster= kmeans_train(float_df, 3)

If we take the first plot we did at the beginning of the post, but now we add the cluster labels, we have this plot.

scatter(filter(row ->row.cluster ==1,df).Social_support, filter(row ->row.cluster ==1,df).Ladder_score, title = "Distribution of Happiness Score by Cluster", xaxis = "Social Support", yaxis="Ladder Score", label = "Cluster 1", legend = :topleft)
scatter!(filter(row ->row.cluster ==3,df).Social_support, filter(row ->row.cluster ==3,df).Ladder_score,  label = "Cluster 2")
scatter!(filter(row ->row.cluster ==2,df).Social_support, filter(row ->row.cluster ==2,df).Ladder_score,  label = "Cluster 3")

Figure 9: Scatter with cluster

Here are the lists in these 3 clusters:

Cluster 1: Australia, Austria, Canada, Denmark, Estonia, Finland, France, Germany, Hong Kong S.A.R. of China, Iceland, Ireland, Luxembourg, Malta, Netherlands, New Zealand, Norway, Singapore, Sweden, Switzerland, United Arab Emirates, United Kingdom, United States, Uzbekistan.

Cluster 2: Albania, Argentina, Armenia, Azerbaijan, Bahrain, Belarus, Belgium, Bolivia, Bosnia and Herzegovina, Brazil, Bulgaria, Chile, China, Colombia, Costa Rica, Croatia, Cyprus, Czech Republic, Dominican Republic, Ecuador, El Salvador, Greece, Guatemala, Honduras, Hungary, Israel, Italy, Jamaica, Japan, Kazakhstan, Kosovo, Kuwait, Kyrgyzstan, Latvia, Libya, Lithuania, Malaysia, Maldives, Mauritius, Mexico, Moldova, Mongolia, Montenegro, Nicaragua, North Cyprus, North Macedonia, Panama, Paraguay, Peru, Philippines, Poland, Portugal, Romania, Russia, Saudi Arabia, Serbia, Slovakia, Slovenia, South Korea, Spain, Taiwan Province of China, Tajikistan, Thailand, Turkey, Turkmenistan, Ukraine, Uruguay, Venezuela, Vietnam.

Cluster 3: Afghanistan, Algeria, Bangladesh, Benin, Botswana, Burkina Faso, Burundi, Cambodia, Cameroon, Chad, Comoros, Congo (Brazzaville), Egypt, Ethiopia, Gabon, Gambia, Georgia, Ghana, Guinea, Haiti, India, Indonesia, Iran, Iraq, Ivory Coast, Jordan, Kenya, Laos, Lebanon, Lesotho, Liberia, Madagascar, Malawi, Mali, Mauritania, Morocco, Mozambique, Myanmar, Namibia, Nepal, Niger, Nigeria, Pakistan, Palestinian Territories, Rwanda, Senegal, Sierra Leone, South Africa, Sri Lanka, Swaziland, Tanzania, Togo, Tunisia, Uganda, Yemen, Zambia, Zimbabwe.

histogram(filter(row ->row.cluster ==1,df).Ladder_score, label = "cluster 1", title = "Distribution of Happiness Score by Cluster", xaxis = "Ladder Score", yaxis="n° countries")
histogram!(filter(row ->row.cluster ==3,df).Ladder_score, label = "cluster 2")
histogram!(filter(row ->row.cluster ==2,df).Ladder_score, label = "cluster 3")

Figure 10: histogram happiness cluster

Finally, we can compare how this cluster affects all the variables.

@df float_df Plots.density(cols();
                             layout=N,
                             size=(1600,1200),
                             title=permutedims(numerical_cols),
                             group = df.cluster,
                             label = false)

Figure 11: Distribution by variables with cluster

3. Conclusions

From my experience using Python for about two years in data analysis and recently dabbling with Julia, I can say that the ecosystem generally seems quite mature for this purpose. I had some questions that the community immediately answered on Julia Discourse. More content like this is needed so that the data science community can more widely adopt this technology.

Finding comfort in the GNU/Linux Desktop

Fri, 19 Jan 2024 00:00:00 +0900

Figure 1: yamete

I have been a Windows user for most of my life, and for a little over two years, I have used MacOS; this article talks about my reasons and experiences in using GNU/Linux as an operating system, the transition, its strengths, and weaknesses.

My first motivation is a mixture of curiosity and the desire to find a free operating system, with the same idea in mind in my previous post where I wanted to move more towards free software, the main piece was the operating system.

For that reason, I have taken two laptops this holiday: a MacBook Pro M1 and an Acer Aspire E5-411. The first is an outstanding Apple laptop; the second is an old machine that I have used at the university since about 2012

1. Ubuntu and Asahi Fedora Remix

For these two machines, I installed different operating systems, for the old laptop at first I wanted to install Arch Linux, the truth is that that operating system attracted me more attention, but I was having problems with the installation and not recognizing the problem, I ended up taking a few steps back and decided to try something simpler. Then, I started my process with Ubuntu.

I still had doubts about the MacBook Pro, as it was the project whose primary choice was to go with Fedora. Even though there are experienced people who have installed another operating system, Asahi's team is working along with this distribution to advance the project of bringing the latest MacBooks so that they can be used without significant problems in Linux environments. But a few days ago, they made the official release. And I decided to give it a chance.

Figure 2: gigachadmachines

The final result: In general, it is better than expected; my old laptop got a second life; I could use it for work that does not require high-intensity computing, such as internet browsing, writing documents, etc. While the MacBook Pro excepting some details still works well.

1.1. Experience with Linux and Asahi Fedora Remix.

To begin with, it's worth highlighting how simple it is to install Linux following Asahi's instructions. In fact, as a user experience, I found it easier to do it through the terminal than through a USB boot, as was the case with Ubuntu. In the installation process, I decided to proceed with Fedora + Gnome for the desktop to maintain a similar standard with the other machine that uses Ubuntu.

What are the problems I mentioned earlier? Primarily the following points:

Bluetooth sometimes has issues connecting devices, and occasionally, you need to delete and reconnect the device, especially with a Logitech mouse that I use frequently.
When entering suspended mode, the machine consumes more battery than MacOS would, which might be the most annoying problem I've encountered
The touchpad has become more sensitive to touch, which can cause your cursor to move while you are typing on the Mac's keyboard.
Audio quality seems to have decreased slightly.

Regarding software, I haven't encountered major issues. I was able to successfully install Codium, Neovim, Alacritty, Tmux, Julia, Python, and R. However, I also wanted to install LibreWolf (a fork of Firefox focused on privacy). Nevertheless, I experienced some problems in the process and decided to abandon that since it wasn't essential software for me.

2. And now what?

Finally, I believe I will continue using Linux, at least for the older PC, as it seems worthwhile to have a supportive device or for simple experimentation. Moreover, my interest in using Linux as the primary operating system is growing. I would probably switch to Fedora since I find the Gnome version of it more user-friendly than Ubuntu's. Regarding the MacBook, I will continue working in the Asahi/Fedora environment, but in case of any complications, I will revert to MacOS. I'll maintain the dual boot, even though I have some specific issues with its interaction. Nevertheless, I still find it quite manageable to use Asahi on this powerful machine.

Set up NeoVim + Tmux for a Data Science Workflow with Julia

Tue, 04 Jul 2023 00:00:00 +0900

Figure 1: love neovim

In this post, I will provide some notes on getting started with Neovim for a Data Science Workflow. This setup is not strictly related to Julia and can also be used with Python and R. The idea is to work with a double panel structure, where one side contains your code and the other side has the REPL, which receives the snippets of code you send from the code side.

In this blog, I will mention the things you can add to make it comfortable for data analysis or more serious development. With this typical kick starter in Neovim and Tmux, I will explain some changes and new packages that are important for this purpose. Finally, I will dive into the details that still need improvement.

1. Why start using Neovim and Tmux? My motivations

I must say, I like notebooks. I used them extensively in my first job in analytics, and they really helped me dive into the problem and experiment with different use cases. I have also worked with VSCode, although I am not a big fan of it, it has helped me in some specific use cases where a more "software engineering" perspective is needed.

With that in mind, when I read the book "Approaching Almost Any Machine Learning Problem" by Abhishek Thakur, and then watched the controversial and yet funny conference by Joel Grus on why he dislikes notebooks, I started to think deeply about the perspective of writing software that follows good practices, is expressive, and still easy to prototype. Unfortunately, I think Grus is right about it. Working with notebooks can lead to some weird behaviors, like running cells in different positions, populating your analysis with too much unnecessary information and plots, not creating abstractions when needed, and having issues with reproducibility. On the other hand, the script perspective didn't help me with fast iteration when I needed quick answers to simple questions.

When I moved to Julia, I realized that the REPL is in another level, and I understood that an important part of this community uses (Neo)Vim or Emacs for development. I was curious about using these tools for data science projects. Although there are not many articles about it, and the community using Vim/Emacs is quite small compared to other options, I found it pretty cool because it is minimalistic (though you can customize it extensively), fast, and promises to increase productivity after mastering the Vim keybindings (in 10 years).

What I realized is that you can still prototype like notebooks (but with a perspective closer to RStudio) with one pane for your code and another pane with your REPL open. Then, you can start transforming your code to make it look like serious software, all within one window without moving all the .ipynb file contents to another .py or .jl file. I found this workflow more enjoyable.

2. Instalation process required

First of all, make sure to install Neovim and Tmux. There are plenty of tutorials out there on this topic, so I won't go into the details here.

The important thing is to create an init.lua file. If you don't want to install everything one by one, I recommend following the kick starter, provided by nvim-lua/kickstart.nvim. It provides the basic tools for working with Neovim, including a package manager, Treesitter, LSP integration, etc. This kickstarter uses LazyVim, which should be faster and doesn't require frequent updates like PackerNvim. Just create the init.lua file, copy and paste all the content into that file, save it, and quit. When you open it again, it should start installing or upgrading everything.

Also you want to create a Tmux config, the default config in Tmux is already ok for working with Data Science or the Julia experience, but anyway you would want to edit a file for setting a colorscheme or edit some shortcuts, to create just add ~/.tmux.conf

You also want to create a Tmux config file. The default config in Tmux is already suitable for working with Data Science or the Julia experience, but you may want to edit it to set a colorscheme or modify some shortcuts. Just add ~/.tmux.conf to create the file.

3. Editing the init.lua and tmux.config

Here are the things required for editing the init.lua file:

Add Julia in the init.lua for treesitter and also consider to add the julials = {} for local servers
Make sure to add the sysimage for the languageserver. in this discussion, they summarize the procedures well, Follow the instructions and add the snippet of code to your init.lua

-- Run Julia LSP
require'lspconfig'.julials.setup{
    on_new_config = function(new_config, _)
        local julia = vim.fn.expand("~/.julia/environments/nvim-lspconfig/bin/julia")
        if require'lspconfig'.util.path.is_file(julia) then
        -- vim.notify("Hello!")
            new_config.cmd[1] = julia
        end
    end
}

This should be enough for setting up Julia. If you open a Julia file, Neovim should be able to detect the LSP and work with other properties like jump to definition, etc. For R and Python, this should be a bit more straightforward for now (not need step 2).
There are other things you want to add, for a data science project, one is vim-slime, This package is great for sending snippets of code from your file to a Julia REPL. Make sure to install it and add the following code to your init.lua. This will allow you to open a Tmux pane and start interacting with the file. You can modify the snippet below to change your target_pane (if you prefer the REPL on your left side or above, you can change this). The actual shortcut is Ctrl-c + Ctrl-c, which you can modify if you prefer.

vim.g.slime_target = 'tmux'
-- vim.g.slime_default_config = {"socket_name" = "default", "target_pane" = "{last}"}
vim.g.slime_default_config = {
  -- Lua doesn't have a string split function!
  socket_name = vim.api.nvim_eval('get(split($TMUX, ","), 0)'),
  target_pane = '{top-right}',
}

For Tmux, there are some things you can add. Here is my config, which is really simple. However, I encourage you to find your own taste with Tmux.

set -g mouse on
set -g history-limit 102400
set -g base-index 1
set -g pane-base-index 1
set -g renumber-windows on

unbind C-b
set -g prefix C-x

# vim key movements between panes
# smart pane switching with awareness of vim splits
bind h select-pane -L
bind j select-pane -D
bind k select-pane -U
bind l select-pane -R
# reloading for now:w

unbind r 
bind r source-file ~/.tmux.conf \; display "Reloaded ~/.tmux.conf"
# plugin
# Initialize TMUX plugin manager (keep this line at the very bottom of tmux.conf)

set -g @plugin 'egel/tmux-gruvbox'
set -g @tmux-gruvbox 'dark' # or 'light'
run '~/.tmux/plugins/tpm/tpm'

For this purpose, I have considered the following in my Tmux config: activate the mouse, increase the history limit in the panes (this is necessary because the default limit in Tmux is quite constrained), count from 1 with panes, change the prefix (I found it easier to use Ctrl-x rather than Ctrl-b), and add the keybindings to move between paneslike vim. The r shortcut is used to restart the config file when you add or modify features, so you can use prefix + r to apply your changes. Finally, in Tmux, you can use a package manager called TPM. Make sure to added in your config file.

4. Some challenges for improving the workflow

So far, the workflow with Neovim and Tmux has been set up nicely. However, there are some areas that can be improved. One of them is the visualization aspect. As a data scientist, you need to constantly iterate and visualize your data. If you want to have a deep understanding of your dataset and generate plenty of visualizations, the current setup may not be the best. However, in Julia, you can easily switch to Pluto to display all the figures you want. One thing I have tried is to constantly display those plots you are working on inside the terminal. One way to do this is by using unicodeplots. If you like working with Plots.jl, you can change your backend from gr() to unicodeplots(). In my opinion, the quality of the visualization may not be the best, but it allows for instant plots in your terminal without the need for third-party software. For fast iteration, it is good enough.

Another important point to consider is maintaining consistency in the workflow between Julia code and the Julia REPL. Currently, I have a workflow with Vim and my code, but the REPL follows a different logic. This is where the aforementioned repository could potentially help, as it aims to bridge the gap and maintain homogeneity between the two panels. Integrating Vim keybindings into the REPL would provide a seamless experience, allowing for a smoother transition and enhancing the overall workflow. It is definitely an area I look forward to exploring in the future to further improve my development process.

5. Conclusions

In this blog, I have explained how to set up Neovim and Tmux with Julia (or any other data science programming language). This setup provides a minimalist perspective. For people who like to have a variety of tools at hand, it may feel a bit lacking. However, if you are someone who is looking for a lightweight tool, minimalistic design, and enjoys working within the terminal, I highly recommend giving it a try.

Sway: The brief introduction to tiling window managers

Wed, 14 Feb 2024 00:00:00 +0900

I can't lie to you that I've had some free time these days, which I've taken advantage of to learn about GNU/Linux Desktop. By installing it on two laptops, I've been able to learn about the kernel and the Ubuntu and Fedora operating systems. However, when I began my journey, I was playing in a safe environment like GNOME, this window manager that provides almost all the tools you need to start without having to tinker too much. But well, certain people say that you also learn about Linux when you have to use a tiling window manager (twm). So, this article is to tell you about how I've used a Mac M1, along with Fedora Asahi Remix, and installing Sway, a popular twm that allows you to create that fancy desktop that you can share on Reddit r/unixporn.

Let's go for the first rice!

Figure 1: kawaii_sway

1. What is a twm? Why is it popular (in Linux)?

It is said that there are two paradigms within the world of window managers: Stacking and Tiling. If you use Windows or MacOS, you may be familiar with the former, where basically the windows you operate for each running software can be overlapped, while the latter works on the avoidance of this (although it can be done) to take advantage of the space used by each program.

Despite this, the popularity of tiling window managers is mainly due to a niche, primarily the Linux community. You can check out the famous Reddit r/unixporn where people customize their operating systems to make them aesthetically and operationally tailored to their users. It's worth noting that this is not exclusively a characteristic of tiling window managers, but they do have a strong representation in that environment.

Finally, the reason why twms are interesting as software is that they are lightweight, highly customizable, and through the use of shortcuts, they can allow your workflow to be more efficient if you learn to use them.

2. Installing Sway and Configuring it.

When installing Asahi Linux on MacOS, I decided to install GNOME, which is a stacking window manager, and I was working normally within that ecosystem. Typically, this environment is quite user-friendly, with LibreOffice installed, Firefox, file management, etc.

Within this environment, I decided to install a tiling window manager (twm). Initially, I wanted to install i3, which is a very popular tiling window manager with more documentation and tutorials available. However, despite my attempts to install it, I ended up failing. This was probably because GNOME was mainly working in Wayland, which is a protocol server that the Asahi Linux ecosystem was based on, while i3 is based on the X11 protocol. For this reason, I had to switch to Sway, which is a twm based on Wayland. To install it, you simply open the terminal and type:

sudo dnf install sway

After entering that command in the terminal, you will install the window manager. Then, you close the session or reboot your system. When you need to log in again, go to the bottom right section and select Sway from the list of window managers to start the new environment. You should notice that the screen has nothing but a new wallpaper and a top bar. From there onwards, you'll need to go to your configuration file located at ~/.config/sway/config and add some configurations if needed. I personally made a few changes, for example:

Adding layouts to switch between Spanish, English, and Japanese:

input type:keyboard {
    xkb_layout latam,us,jp
    xkb_options grp:alt_shift_toggle
}

Tab style, where I remove borders, add gaps, and use a color adjusted to the gruvbox format for focused windows:

gaps inner 10
gaps outer 10

client.focused "#98971A" "#98971A" "#ffffff" "#98971A"
font pango:monospace 0
titlebar_padding 1
titlebar_border_thickness 0

Set Audio and brightness shortcuts

#Set audio 
# Increase volume
bindsym $mod+F12 exec pactl set-sink-volume @DEFAULT_SINK@ +5%

# Decrease volume
bindsym $mod+F11 exec pactl set-sink-volume @DEFAULT_SINK@ -5%

# Mute/unmute
bindsym $mod+F10 exec pactl set-sink-mute @DEFAULT_SINK@ toggle

#Set Brightness
# F9 - Decrease brightness
bindsym $mod+F9 exec brightnessctl s 5%+

# F8 - Increase brightness
bindsym $mod+F8 exec brightnessctl s +5%-

This is generally some of the changes I made, trying to adjust it to my workflow, and it's been going quite well, to be honest.

3. shortcuts shortcuts shortcuts!

One of the strengths (and challenges) of working in a twm is that you quickly feel encouraged to use more shortcuts after realizing that you can open the terminal, the browser, send windows to other workspaces, and move between those workspaces. If you're using the Vimium extension in your browser and also use Vim for text editing, you may resonate with this way of working as it's quite consistent.

I get the impression that twms strongly encourage working in the terminal, so you may consider installing other software to work harmoniously in this environment. Some of the software I've been incorporating into this environment:

(Neo)Vim: Text editor
Tmux: Terminal multiplexer
Newsboat: An RSS reader
Alacritty: Terminal
Zsh: Unix shell
Cmus: Music player
Ranger: File manager
htop: For system monitoring
waybar: A highly customizable bar for Sway
dmenu: Application launcher

You can visit my .dotfiles

4. Is it all worth it?

Short and obvious answer: it depends. If you're someone willing to invest the time to understand and configure your system, I'm sure it can accelerate your workflow process, and you might even enjoy working in this mode. I personally have enjoyed it, especially the way you can simply fly between different workspaces, making the experience incredibly satisfying. However, I've also had some annoying moments. For example, when you want to connect your devices via Bluetooth, you'll have to do it in the terminal, although you can open the settings within Sway if you're using GNOME or KDE Plasma, but otherwise you'll have to figure out how to do it in the terminal. Similarly, if you want to connect to Wi-Fi or take a screenshot, these small things that are so simple in general in other systems with a GUI can be a bit annoying to perform in the terminal or try to build your own shortcut in a config file. So, things that you had very well resolved before, you now have to rethink.

Likewise, if you're not someone who is really interested in delving into config files and prefers just navigating with your mouse within a GUI, then simply stick with GNOME or KDE or any other stacking window manager. Although I would still recommend installing a twm at least once to experience it.

Experiencias con el método Zettelkasten e Emacs

Sun, 02 Feb 2025 00:00:00 +0900

Durante un par de semanas incorporé una nueva herramienta en Emacs y en mi "workshop digital", he estado usando la herramienta Denote y probando el método Zettelkasten en mi día a día, aquí compartiré algunas experiencias sobre este método, tal vez te ayude a ti a ver si te interesa incorporar este método en tu día a día.

1. ¿Qué es el método Zettelkasten?

Para ir directo al grano el método tiene un proposito, que es construír una compleja red de conocimiento personal y que este sea facil de administrar. Esta red se basa en "nodos" de conocimiento que sean lo más atómicos posibles y construir "links" entre estos nodos para cohesionar el contenido de tal forma que puedas ir enlazandolo en la red, estos nodos contendrán una metadata referenciando tópicos, fechas, títulos, links, etc. Con esto el conocimiento es capturado, puede ser conectado, cohesionado y administrado.

El método puede ser creado (notas, carpetas, post-it, etc ), como también usado con herramientas digitales, existe un montón de software ahí afuera que trata de cubrir estas propiedades como Obsidian, Forester, Mem, Denote, etc. Antes de que trates de usar algunos de estos softwares, pensemos un poco en cual es el ejercicio que tú harías en tu laptop si quisieras usar esta metodología.

2. Aplicación de la metodología

Imagina que estás estudiando inglés en tu día a día y te gustaría trackear tu conocimiento sobre el lenguaje, y la forma en que quieres tomar notas en una carpeta llamada Zettelkasten donde estas creando distintos archivos, principalmente en .txt (aunque podría ser cualquier tipo de archivo). Hoy aprendiste una nueva regla gramatical, y quieres guardar ese conocimiento, entonces harás algo como lo siguiente:

Ir a la carpeta Zettelkasten, y crear un archivo .txt con la estructura Fecha--Titulo__Tag.txt, imaginemos que lo llamas 20250130t105229--uso-de-la-palabra-more-y-less__ingles_gramatica.txt
Escribirás como leíste en el título a lo mejos escribirás una breve definición y ejemplos sobre estas palabras como:

- more significa más y less significa menos
  ejemplo: I love dogs more than cats.
  I feel like I have less time these days

Luego guarda ese archivo, y continua con tu día.

Hasta acá hay algunas cosas especiales, como puedes ver el archivo creado es un nodo de conocimiento y su título contiene metadata (fecha, nombre, tópicos), y su contenido por supuesto hasta ahora puede que sea una simple toma de notas, pero en la medida de que vas incorporando nuevo conocimiento, puedes construir relaciones y agregar nuevas capas de conocimiento. Imaginemos que aprendiste algo nuevo en inglés, y tomas notas sobre esto:

Crear un archivo 20250211t221209--regla-consecuencia-the-more-the-more__ingles_gramatica.txt
Escribes sobre la regla the more.. the more

Cuando quieres usar un efecto que se produce cuando algo cambia puedes
usar la regla the more... the more

ej: - the more you study, the more you know
- the less time I have, the more productive I am

Importante saber las reglas gramaticales more o less para usar esta regla,
revisar 20250130t105229--uso-de-la-palabra-more-y-less__ingles_gramatica.txt

Guardar el archivo y seguir con tu vida.

Como puedes ver aquí volví a crear un archivo y esta vez linkié este conocimiento con otro nodo. Así que con esto podemos ir creando una carpeta de conocimientos y la metadata nos ayuda a preservar clasificaciones, por lo que si quisieras también estudiar alemán por ejemplo, puedes aprovechar esto para conectar conocimiento, tal vez se te ocurra crear un nodo (.txt) para comparar gramática en ingles y aleman, o tal vez estas también estudiando economía, y quieres crear una tarjeta que incluya terminos económicos en inglés, por lo que usaras tags como "economía" e "inglés", por ejemplo.

Es bastante simple, y no requieres ser un ingeniero de la Nasa para construir algo como esto, puedes además facilmente buscar y filtrar archivos a través de cualquier search, organizarlo y depurar el conocimiento en la medida que accedes a él. Aquí coloqué ejemplos usando .txt, pero podría ser lo que sea: archivos, imágenes, .pdf, videos, etc. Así mismo puedes agregar links a la web, entre otras cosas.

Probablemente lo más tedioso en usar este método es como automatizar el proceso de creación, guardado y linkeo de estos nodos, y de ahí que los softwares pueden facilitar bastante este proceso.

3. Usando Emacs y Denote

El flujo de trabajo anterior es una explicación práctica (y opinable) de como he estado usando Denote para el método Zettelkasten, obviamente Denote me ayuda en la automatización de este proceso. Cabe notar de que otra alternativa bastante popular a Denote es OrgRoam, dado que era más popular me interesaba más, pero la idea de setear una base de datos para poder trabajar con archivos, si bien práctica considero que le agrega una capa de complejidad innecesaria. Por lo que decidí testear Denote primero y terminó siendo más que aceptable.

En mi init.el agregué lo siguiente:

;; Denote to taking notes or the Second Brain
(use-package denote
  :ensure t
  :custom
  (denote-directory "~/Denotes")
  (denote-rename-confirmations '(rewrite-front-matter))
  (denote-save-buffers t)
  :config
  (add-hook 'text-mode-hook #'denote-fontify-links-mode-maybe))

;; Integration Denote with Org capture

(with-eval-after-load 'org-capture
  (add-to-list 'org-capture-templates
               '("n" "New (De)note" plain
                 (file denote-last-path)
                 #'denote-org-capture
                 :no-save t
                 :immediate-finish nil
                 :kill-buffer t
                 :jump-to-captured t)))

No hay mucho que agregar acá más que la instalación de la librería y asegurarme de que en cualquier momento que precione C-c c pueda abrir org capture y seleccionar Denote como opción, se abre un prompt para agregar el título y tags. Y de inmediato se abre un archivo .org con la metadata lista, y puedo tranquilamente escribir lo que requiera, luego salvar el archivo con C-c C-c, y volver a lo que estaba haciendo previamente. También puedo linkear notas para empezar a relacionar conceptos dentro de la red de notas, y así esta se va ir desarrollando organicamente.

Quiero recalcar que Denote es una herramienta con mayor complejidad, tiene bastantes funcionalidades, pero mi uso es relativamente sencillo, por lo que no he sentido la necesidad de experimentar con todo lo que puede ofrecer esta herramienta.

4. Conclusiones

En estas semanas experimentando con el método Zettelkasten he visto varias cosas interesantes, el primero es que la idea de tomar breves periodos de estudios del tópico que te interesa, se hace más sencillo de desarrollar organicamente con un sistema de toma de notas instantaneas que vas depositando en un repositorio, esto es ideal cuando te encuentras en el proceso de que aprendes algo mientras vas de paso. Pero algunos desafíos que tienes que considerar:

Zettelkasten debiera ser un espacio para escribir notas depuradas de lo que has aprendido, no algo como un copy paste de una definición de Wikipedia.
Es positivo revisar estas notas constantemente, porque te permite retroalimentarte de ese conocimiento y depurarlo mientras desarrollas nuevas ideas.
Esto no es algo para escribir números de telefonos o lista de supermercado
La parte más importante es sobre la producción de ideas y la conexión de estas para la producción de nuevo conocimiento
La automatización ayuda bastante para que la producción de la red sea más llevadera, esto te permite capturar ideas y conocimientos de forma instantanea.

5. Referencias

He decidido retirarme del Fediverso

Mon, 07 Jul 2025 00:00:00 +0900

Update 2025-07-09: Agregado unos links de otros artículos y edición de ciertas lineas.

Desde hace unos meses terminé de convencerme de que el Fediverso, pero en especial el uso de los miniblogs (Pleroma, Mastodon, etc) no eran para mi y he decidido eliminar mi cuenta, probablemente si leiste mi post de sobre como migrando a Mastodon me sirvió para conocer una subcultura de geeks y hackers que me inspiraron a insertarme más en el mundo del software libre, si bien todavía mantengo mi apreciación por la experiencia, he decidido continuar mi camino por otras sendas.

Por un periodo de tiempo, estuve defendiendo el fediverso como una mejor alternativa a Twitter y otras plataformas centralizadas, y la verdad, todavía creo que son mejores y a cualquiera que este cansado de las big tech, le diría de que podría probar un tiempo en Mastodon o Lemmy y tal vez logre encajar. Pero para mi esa mejora en mi condición de vida, fue solo un paso previo para simplemente no usar más las redes sociales y aquí quiero proponer algunas razones del por qué.

Si en Twitter desde sus inicios se caracterizaba por ser un nido de cuervos tratando de obtener beneficios incitando al enojo de los usuarios a través de polémicas andrajosas y el clickbait, la verdad es que con cierta moderación las plataformas federadas también tienen una buena dosis de violencia, es solo la tortilla dada vuelta, solo que un poco menos tostada por la carencia de features que ofrece la plataforma, lo cual ayuda a sanitizar un poco las interacciones entre los usuarios. Pero así como en Twitter tal vez la comunidad LGBTQ+ puede sentirse en peligro, no esta exagerado decir de que en el Fediverso encontrarás bastante comentarios de odios para gente de derecha, conservadores, personas que siguen una religión, etc. He tenido que bloquear o reportar a usuarios que han posteado comentarios bastante desagradables.

Además de esto he visto como el Fediverso ha actuado en efecto de enjambre para purgar ciertas personalidades "indeseadas", generando que instancias completas bloqueen a la instancia que aloja a dicho usuario afectando también a los demás usuarios que poco o nada tienen que ver, me acuerdo que en una crítica que hice al respecto un usuario argumentó que la culpa era finalmente de los administradores que aceptaban a ese tipo de "indeseables" en las instancias, y ¿qué era lo que hacía esta persona indeseable?, bueno, era un Youtuber que hablaba de software libre, pero con tendencias más libertarias y/o de derecha que la mediana del espectro político del fediverso. En su momento no encontré grandes razones para esa caza de brujas.

Esto por supuesto no es el único caso, ya que puedes encontrar un gran número de personas que no han tenido un recibimiento tan agradable dentro de la plataforma por distintos méritos, aquí hay la experiencia de un usuario con el que he interactuado en la web ¹ :

"… My personal experience but I recall when trying to find academic Mastodon servers, they expressly prohibited potential members who had worked with, whether through contracting or otherwise, academics who had interaction with US government entities like the US DoD. Since my research work had received some funding from these groups, I was disqualified. I saw this repeated in a few avenues and, well, I just gave up on Mastodon servers and landed on Bluesky which made me feel more welcomed…"

También Drew Devault, uno de los desarrolladores de software libre detrás de proyectos como Sway, que en su momento participaba activamente en el Fediverso comentó al respecto en su blog ² :

"Social networks are not good for you. The Fediverse brought out the worst in me, and it can bring out the worst in you, too. The behaviors it encourages are plainly defined as harassment, a behavior which is not unique to any ideological condition. People get hurt on the Fediverse. Keep that in mind."

Muchos de los vicios de las redes sociales mainstream también estan presentes en estas plataformas descentralizadas como pueden ver. Por mucho que queramos vender que el Fediverso es un oceano de paz a su contraparte Twitter (X), la verdad es que me es dificil defender esa posición.

2. Carencia en la confianbilidad de la continua operación de las instancias

Otro problema que tengo con el fediverso es la confiabilidad en la persistencia de las instancias, no en la red en cuestión, o que Activity Pub vaya a colapsar ni nada de eso, pero que debes tener una confianza en que el administrador XX de tu instancia haga el trabajo de mantener el servidor, y que no desaparezca de un momento a otro. Estuve por bastante tiempo en Fosstodon, y puedo dar fe de que era una buena instancia y agradezco a las personas que se dieron el trabajo de mantenerla, sé que no era un trabajo fácil, yo por otro lado quise probar un poco self-hosteando una instancia de Pleroma en mi servidor personal, y hay que reconocer que el trabajo que hay detrás de los servidores y filtrado de la comunidad es un tremendo trabajo, no es raro pensar que instancias a veces desaparecen de un momento a otro porque el administrador dejó de preocuparse por ella. Un caso particular que me tocó ver de cerca fue Julialang.social, donde un momento a otro, la instancia desapareció, sin previo aviso, y todos los usuarios terminaron siendo afectados perdiendo de forma inmediata su cuenta. Usualmente la regla de oro que se tiene en el Fediverso es no hacerse usuario de una instancia pequeña y poco conocida, pero ir a una instancia popular (como Mastodon.social, Fosstodon, etc). Pero nuevamente nos encontramos con el problema de concentración.

Parte de tener una instancia personal es en mi opinión más simple y ayuda a que la red se estabilice al depender menos de mega instancias, pero esto me hace pensar que la simplicidad importa y que conectar con la gente a través de las formulas más sencillas posibles es menester. No través de big data centers de las empresas tecnológicas, ni tampoco de aglomeraciones de servidores medianos descentralizados con personas teniendo que realizar trabajos de administrador de sistemas de manera voluntaria en su tiempo libre, sino una relación donde cada uno se debe preocupar por su propia "instancia".³

3. La solución es volver a los sitios web personales y blogs

Francamente, he tenido un mayor disfrute leyendo blogs de gente interesante que de la gran mayoría del tiempo que gasté interactuando con personas en redes sociales, actualmente uso elfeed (Rss reader de Emacs) para seguir algunas blogs y noticias, también el hecho de que no tengo restricciones para escribir, me permite darme el tiempo de pensar y elaborar más lo que voy a decir, es el acto de pensar en cuestión y razonar de forma más elaborada tus ideas de lo que te puede entregar un sitio donde tienes limitado a 200 carácteres o menos.

Mi website no tiene botones de interacciones, ni likes, ni sección de comentarios, si alguien quiere hablarme puede escribirme al correo, y podemos seguir la discusión (tal vez abra una instancia de chat en XMPP o similar), con esto puedo reducir los incentivos para interactuar en un estado donde emociones negativas sean el motor de la conversación.

Actualmente estoy usando Franklin.jl para generar los markdown a html, mantener archivos html es de costo mínimo, nada de mantener bases de datos, si no tengo un servidor puedo hostear en un vps o Github pages, es probablemente una de las soluciones más sencillas hechas así que no tengo que sufrir por actualizar nuevas versiones del software, migrar bases de datos, usar Docker, etc.

4. Conclusiones

Con esto, no quiero decir que el Fediverso es terrible y que hay que hacerlo desaparecer, si hay alguien que esta cansado de las Big tech y quiere otra cosa similar, recomiendo que se vaya a Mastodon.

Pero aún así creo que el gran punto final es volver a lo básico, ten tu website, ajusta tu RSS Feed (¡Por favor con texto completo a disposición!) y deja que tus ideas sean guíadas como una hoja en blanco sobre un escritorio y no que tu mente sea estrujada en una especie de coliseo romano donde lo que quieres es que el publico te regale su pulgar arriba para seguir opinando sin miedo.

5. Referencias

Footnotes:

Early Impressions of Bluesky

The Fediverse can be pretty toxic

Comparto con este artículo

From Mastodon to Free software: My little experience

Thu, 21 Sep 2023 00:00:00 +0900

Figure 1: chadfree

1. Introduction

It has been a while since I decided to purge several of my social networks (including Twitter, Facebook, Instagram, and LinkedIn) and mainly stick to Mastodon as the only platform for interacting with people on the internet whom I don't really know. So far, it has been a positive experience, and I have no doubts that it has improved my quality of life. Furthermore, it has opened up a world that I wasn't very aware of and that I have become increasingly interested in: The world of Free software.

2. First Steps

My first real interactions with the world of open source (which was the term I was familiar with) were through my work as a data analyst at one of the companies I worked for. I used Python for most of my analyses because I felt I could do more than the average business analyst with Excel. For the first time, I got involved with the term open source, and I was amazed that there were people who donated some of their time to create projects to share with others (of course, this is a generalization, understanding that there are also various types of organizations). Even more so, someone like me could use these libraries and benefit greatly from them in advancing my projects for which I was paid.

I believe that this is where I began to realize the power of community work and how these projects were not necessarily something artisanal that easily broke. On the contrary, many of these projects are quite solid and provide a great platform for anyone with basic programming skills to use and derive significant value from.

3. The Breakup with Closed Software Begins

To be honest, this breakup started with social media, which I was an avid user of at the time, and for some reason, I felt quite tired of them. However, I still wasted my time on them. Twitter, in particular, had me fed up because of the level of aggression and toxicity on the platform. When I wanted to read about a topic related to my country, I ended up feeling annoyed by how the interactions were happening (which was similar to the discussions among family members with different political views at the annual family dinner). This reached a breaking point when Elon Musk took over Twitter. Why? Because I never believed in his idea of free speech and his intentions to make Twitter a "better place." Time has shown us that EM has never been a man of science, and his altruistic purposes are nothing but a nebula of contradictions.

3.1. A Wild Mastodon Appears

Figure 2: mastodon

With my intention to leave Twitter, I learned about the alternative of Mastodon as a decentralized platform that is "similar to Twitter," where privacy was the most important part of its speech. I decided to create an account and see how it was. In fact, I chose Fosstodon (an instance whose main focus is free and open-source software), which is strange because I'm not a software engineer, nor was I actively participating in any FOSS project. My first impressions: absolutely boring. I won't lie; initially, I didn't understand the dynamics of choosing an instance and didn't understand the platform's trending topics. As I was part of the wave of people who decided to leave Twitter, there were several new accounts on Mastodon, and the trend of leaving the birdsite was strong in those days. But then things normalized, and it all seemed more relaxed. The problem was that I kept going back to Twitter because I didn't connect with Mastodon, and my browsing on Mastodon ended in 10 minutes. I returned to Twitter, where everything was "buzzing," and there was always a controversy to engage with.

This behavior persisted for several weeks, but my desire to get rid of Twitter became stronger. And finally, I deleted my account, which I have no regrets about. The truth is that I felt more relaxed, spent a little less time on my phone, realized that many of the influencers and experts in the topics that interested me were posting a lot of irrelevant content. When there was an important trend, I could discover it through other means (on Mastodon, Reddit, or one of the blogs I follow). That's how I began to embrace the idea of staying on Mastodon, where everything was quieter for various reasons:

No (tricky) algorithms: This is something that really bothers me about social networks. Platforms try to create recommendation systems to keep you hooked, and social media have embraced this method to keep you there. It's incredible how these algorithms can influence your thinking and what you consume. So, seeing that your timeline is simply organized chronologically is really refreshing.
Privacy: With the first point, you understand the impact of giving away your data to an organization that can do whatever it wants with it, from selling the data to other organizations to training its machine learning algorithms. Our problem is that we have been poorly educated about the value of protecting our personal information, and it's not the same to easily give it away to any institution, website, or anything else.
More optimized for conversation: Honestly, I think I've seen better discussions in general on Mastodon than on other social networks. It's not that these discussions don't exist on other platforms, but they are a tiny fraction beneath a mass of people trying to sell content. You can notice on LinkedIn the influencers who use the term "game changer" every week to describe the next technology or software they discovered. On Twitter, people share bullet points in a thread about "how to basics," which is nothing more than a reuse of content someone else posted a few days ago. Or the idea of delivering the simplest and most effective message that gets the best ratings, but it's full of fundamental errors, which is quite typical in political discussions. On Mastodon, these interactions can also happen, but there's no algorithm boosting this type of content, so it's almost nonexistent. This ends up discouraging all kinds of influencers looking to reach more people with certain strategies, creating a parasocial effect as mentioned in the blog of Drew Devault, which I recommend reading other articles from that blog.

3.2. A Space to Understand the Impact of Free Software

Over time on Mastodon, I began to read posts from people in the tech world who weren't just occasional users of open-source software (as I am), but actively participated in its improvement and promotion. I understood why many of these people reduced their use of social networks and, instead, had their own blogs (in fact, this blog started with that idea in mind). And the idea of reclaiming personal space on the internet instead of depending on oligopolies and other companies with questionable practices.

Over time, I started evaluating the software I use and the websites I visit to reduce the impact they might have on matters related to my privacy. I ended up deleting other social networks and looking for open-source software alternatives or at least ones that have a more respectful attitude toward privacy for things I used to do with proprietary software. Here are some of the changes I've made (note that these are not the only alternatives, so I recommend researching if you don't like these software choices):

Excel -> Python, R, Julia
Word, PowerPoint -> Typst, Pandoc, Quarto
GitHub -> Codeberg
VSCode -> Vim/Neovim
Chrome -> Firefox
Google Search -> DuckDuck Go
Twitter -> Mastodon
Medium, Substack -> This blog and use of an RSS reader (newsboat) to read other blogs
Mac OS -> Considering a move to Asahi Linux
Nord Pass -> Bitwarden

And I'm still making changes…

But I haven't been able to make all the changes I wanted. For example, for staying in touch with my family from a distance, I still use WhatsApp. Obviously, this is a sensitive issue that is not negotiable, so as long as I don't have an alternative, I will continue to use that application. However, I may use Signal as a replacement in the future.

5. What Positive Impacts Have You Had?

The most important and highlighted impact is the feeling of tranquility and less stress. After purging social networks (Twitter, Facebook, Instagram, and LinkedIn), I'm simply much less attached to my phone, and I receive less stress from information overload. It's basically one less addiction in my life, and it's difficult to control. If you're the person who's glued to your phone, it's not your fault, but it's these platforms that turn their product into a drug to keep you there for as long as possible. In fact, checking the statistics of my Instagram account, I spent around 1 to 2 hours a day there. Doing what? Nothing relevant, if you ask me. So, I won't deny that my life has improved.

On the other hand, I've learned different things about computers that I had no idea about before, and I've become better at it. Just by knowing certain tools, I became more proficient at work, so I could automate things that other colleagues simply didn't know how to do. I automated tasks with Bash scripting, created pipelines with Python to manipulate data, and learned a bit more about statistics since I now had libraries available, among other things.

Lastly, I've learned more about the value of personal information and independence. Many of the proprietary software we use have some form of telemetry or seek to gather all kinds of information about you. I understand that people don't place much value on their personal information, but I think it's something we shouldn't so easily give away. After all, companies make money directly or indirectly from your information, and it's a shame that schools and the education system, which are supposed to be guardians of forming free and independent individuals capable of thinking for themselves, do not emphasize these topics and promote dependence on proprietary software without even mentioning these alternatives. For example, when Windows forces you to accept new updates or changes to the license (which few probably read), you can realize that you're not really free to choose. And if your PC becomes slow because software updates are denser than any hardware can handle, then consider installing Linux and saving that PC.

6. What Negative Impacts Have You Had?

The truth is the only negative impact I can mention is that by not using social networks, I'm not "up to date" with what acquaintances and colleagues are doing. I don't know if they changed jobs or what they did yesterday, which was something Instagram, for example, was good at. Is it really important? Not at all. In fact, I think we tend to overvalue the idea that knowing in real-time what's happening with 300 people (or more) is important. I have a group of friends with whom I still talk from time to time, and I can really know what they're doing, and that's enough for me. If there are people who matter to you, ask them for their email or some other means of contact where you can chat from time to time.

7. Conclusions

This has been my experience primarily delving into the world of open-source software and reducing the consumption of social networks. These are two different things but are closely related. To summarize in a few points:

Try to eliminate social networks, especially those supported by recommendation systems to keep you hooked.
Consider and value the idea of using open-source software. If you're using proprietary software, evaluate if there's an alternative that upholds the fundamental principles of open-source software or at least follows minimum values of security and privacy.
Learn more about open-source software, and if there's software you use regularly that helps you in your daily life, look for ways to support it.
Don't be afraid to learn more. I believe the idea of "I don't know about computers," "I'm not a programmer," among others, is a poor argument for self-limitation. Take the time to learn gradually about these topics.

I hope this has been helpful, and if you have any questions, you can write to my email or find me on Mastodon.

Using evotrees.jl for time series prediction

Thu, 02 Mar 2023 00:00:00 +0900

1. Introduction

In this post, I want to show an analysis of a time series that I've been working on. Usually, when dealing with time series, it is not so common to use machine learning algorithms (without at least trying more traditional models like the ARIMA family), but I still wanted to test how well a GBM model fits for these kinds of problems that are so popular.

NOTE: I don't recommend starting with models of this type for time series problems. There are simpler models to understand that are less computationally expensive.

2. Dataset Preparation

2.1. Data Extraction

You can find the repository here, The codes you will see here, I prototyped in notebooks/tutorial.jl.

Now we start by making the corresponding imports.

using DataFrames
using Plots
using MLJ
using EvoTrees
using UrlDownload
using ZipFile
using HTTP
using CSV
using Dates
using Statistics
using MLJClusteringInterface
using Clustering
using FreqTables
using StatsPlots
using RollingFunctions
using StatsBase
using ShiftedArrays

There are several libraries in this section, and I must admit it took me some time to use each one. But anyway to start reading the dataframe, we can get it directly from its repository.

data_url = "https://archive.ics.uci.edu/ml/machine-learning-databases/00235/household_power_consumption.zip"
f = download(data_url)
z = ZipFile.Reader(f)
z_by_filename = Dict( f.name => f for f in z.files)
data = CSV.read(z_by_filename["household_power_consumption.txt"], DataFrame,)

The dataframe looks more or less like this:

     Row │ Date        Time      Global_active_power  Global_reactive_power  Voltage  Global_intensity  Sub_metering_1  Sub_metering_2  Sub_metering_3
         │ String15    Time      String7              String7                String7  String7           String7         String7         Float64?
─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       1 │ 16/12/2006  17:24:00  4.216                0.418                  234.840  18.400            0.000           1.000                     17.0
       2 │ 16/12/2006  17:25:00  5.360                0.436                  233.630  23.000            0.000           1.000                     16.0
       3 │ 16/12/2006  17:26:00  5.374                0.498                  233.290  23.000            0.000           2.000                     17.0
       4 │ 16/12/2006  17:27:00  5.388                0.502                  233.740  23.000            0.000           1.000                     17.0
       5 │ 16/12/2006  17:28:00  3.666                0.528                  235.680  15.800            0.000           1.000                     17.0
       6 │ 16/12/2006  17:29:00  3.520                0.522                  235.020  15.000            0.000           2.000                     17.0
       7 │ 16/12/2006  17:30:00  3.702                0.520                  235.090  15.800            0.000           1.000                     17.0
       8 │ 16/12/2006  17:31:00  3.700                0.520                  235.220  15.800            0.000           1.000                     17.0
       9 │ 16/12/2006  17:32:00  3.668                0.510                  233.990  15.800            0.000           1.000                     17.0
      10 │ 16/12/2006  17:33:00  3.662                0.510                  233.860  15.800            0.000           2.000                     16.0
      11 │ 16/12/2006  17:34:00  4.448                0.498                  232.860  19.600            0.000           1.000                     17.0
      12 │ 16/12/2006  17:35:00  5.412                0.470                  232.780  23.200            0.000           1.000                     17.0
      13 │ 16/12/2006  17:36:00  5.224                0.478                  232.990  22.400            0.000           1.000                     16.0
      14 │ 16/12/2006  17:37:00  5.268                0.398                  232.910  22.600            0.000           2.000                     17.0
    ⋮    │     ⋮          ⋮               ⋮                     ⋮               ⋮            ⋮                ⋮               ⋮               ⋮
 2075246 │ 26/11/2010  20:49:00  0.948                0.000                  238.160  4.000             0.000           1.000                      0.0
 2075247 │ 26/11/2010  20:50:00  1.198                0.128                  238.110  5.000             0.000           1.000                      0.0
 2075248 │ 26/11/2010  20:51:00  1.024                0.106                  238.840  4.200             0.000           1.000                      0.0
 2075249 │ 26/11/2010  20:52:00  0.946                0.000                  239.050  4.000             0.000           0.000                      0.0
 2075250 │ 26/11/2010  20:53:00  0.944                0.000                  238.720  4.000             0.000           0.000                      0.0
 2075251 │ 26/11/2010  20:54:00  0.946                0.000                  239.310  4.000             0.000           0.000                      0.0
 2075252 │ 26/11/2010  20:55:00  0.946                0.000                  239.740  4.000             0.000           0.000                      0.0
 2075253 │ 26/11/2010  20:56:00  0.942                0.000                  239.410  4.000             0.000           0.000                      0.0
 2075254 │ 26/11/2010  20:57:00  0.946                0.000                  240.330  4.000             0.000           0.000                      0.0
 2075255 │ 26/11/2010  20:58:00  0.946                0.000                  240.430  4.000             0.000           0.000                      0.0
 2075256 │ 26/11/2010  20:59:00  0.944                0.000                  240.000  4.000             0.000           0.000                      0.0
 2075257 │ 26/11/2010  21:00:00  0.938                0.000                  239.820  3.800             0.000           0.000                      0.0
 2075258 │ 26/11/2010  21:01:00  0.934                0.000                  239.700  3.800             0.000           0.000                      0.0
 2075259 │ 26/11/2010  21:02:00  0.932                0.000                  239.550  3.800             0.000           0.000                      0.0
                                                                                                                                   2075231 rows omitted

2.2. Dataset Cleaning

As can be seen, it is a quite large dataset and we can take the opportunity to create new variables, so we have the possibility to obtain relevant information.

#Create a variable 
date_time = [DateTime(d, t) for (d,t) in zip(data[!,1], data[!,2])]

data[!,:date_time] = date_time

#Create variable for date
data[!,:year] = Dates.value.(Year.(data[!,1]))
data[!,:month] = Dates.value.(Month.(data[!,1]))
data[!,:day] = Dates.value.(Day.(data[!,1]))

#Create variable for time
data[!, :hour] = Dates.value.(Hour.(data[!,2]))
data[!, :minute] = Dates.value.(Minute.(data[!,2]))

#Create variable for weekends
data[!, :dayofweek] = [dayofweek(date) for date in data.Date]
data[!, :weekend] = [day in [6, 7] for day in data.dayofweek]

In addition, we notice that the variables are in String format. We can make some changes to put them in the appropriate form.

for i in 3:8
    data[!,i] = parse.(Float64, data[!,i])
end
data[!,1] = replace.(data[!,1], "/" => "-")
data[!,1] = Date.(data[!,1], "d-m-y")

3. Preliminary Visualizations

A classic way to plot all the variables is with the following code:

plot([plot(data[1:50000, :date_time],data[1:50000,col]; label = col, xrot=30) for col in ["Global_active_power",  "Global_reactive_power", "Global_intensity", "Voltage", "Sub_metering_1",  "Sub_metering_2", "Sub_metering_3"]]...)

Figure 1: Line Plot All

Note that we only take a sample of 50,000 data points to avoid overloading the graphs with information, and in the same way, we can create histograms.

plot([histogram(data[1:50000, col],label = col, bins = 20 ) for col in ["Global_active_power",  "Global_reactive_power", "Global_intensity", "Voltage", "Sub_metering_1",  "Sub_metering_2", "Sub_metering_3"]]...)

Figure 2: Line Plot All

For now, we can recognize that the time series in its global variables have a white noise behavior, and Voltage also has it, however, it is the only one that seems to have a distribution that is similar to a normal distribution, while the sub-metering, are signs of use of household appliances.

4. A brief clustering with kmeans

In this section, we are interested in building a clustering model on the time series. The purpose? It is simply a way of evaluating behavior patterns over time, one hypothesis would be to see irregular behavior patterns over time, given that greater consumption would be seen at specific periods of the day or season.

An interesting issue that I was unaware of was that time series clustering is possible and you can use k-means, however in these cases, they cannot be treated from the same perspective, and other types of variants of these algorithms should be used to consider the temporality of neighboring observations when clustering. But since this project is just a toy, and the use of this technique is only for EDA, we will stick with the classical algorithm.

If you want to know more about this topic, yu can read this articule

Continuing with the problem, we can cluster by applying the following code.

X = data[!, 3:9]
transformer_instance = Standardizer()
transformer_model = machine(transformer_instance, X)
fit!(transformer_model)
X = MLJ.transform(transformer_model, X);
KMeans= @load KMeans pkg=Clustering
kmeans = KMeans(k=3)

mach = machine(kmeans, X) |> fit!

# cluster X into 3 clusters using K-means
Xsmall = MLJ.transform(mach);
selectrows(Xsmall, 1:4) |> pretty
yhat = MLJ.predict(mach)
data[!,:cluster] = yhat

In this case, we have 3 clusters that are ordered as follows.

cluster nrow
CategoricalValue    Int64
1   1   741077
2   2   1257309
3   3   50894

And if we try to plot the clusters, we would have the following.

plot([scatter(data[1:20000, :date_time],data[1:20000,col]; group=data[1:20000,:].cluster, size=(1200, 1000), title = col, xrot=30) for col in ["Global_active_power",  "Global_reactive_power", "Global_intensity", "Voltage", "Sub_metering_1",  "Sub_metering_2", "Sub_metering_3"]]...)

Figure 3: scatter plot cluster

It looks a bit confusing, although if we look at the voltage variable, we can already size up a certain trend. For now, let's consider a boxplot of the main variables but considering the clusters.

b1 =@df data boxplot(string.(:cluster), :Global_active_power, fillalpha=0.75, linewidth=2, title ="Global active power")
b2 =@df data boxplot(string.(:cluster), :Global_reactive_power, fillalpha=0.75, linewidth=2, title = "Global reactive power")
b3 = @df data boxplot(string.(:cluster), :Global_intensity, fillalpha=0.75, linewidth=2, title ="Global intensity")
b4 = @df data boxplot(string.(:cluster), :Voltage, fillalpha=0.75, linewidth=2, title = "Voltage")


plot(b1, b2, b3, b4 ,layout=(2,2), legend=false)

Figure 4: scatter plot cluster

The truth is that we notice slight differences between the clusters, where we have certain consumption patterns in each category, but in some of their variables these do not necessarily lead us to any conclusion. However, as we had mentioned at the beginning, the idea of clustering was to study consumption patterns during time intervals, so we add the following.

h1 =heatmap(freqtable(data,:cluster,:dayofweek)./freqtable(data,:cluster), title = "day of week")
h2 =heatmap(freqtable(data,:cluster,:hour)./freqtable(data,:cluster), title = "hour")
h3 = heatmap(freqtable(data,:cluster,:month)./freqtable(data,:cluster), title = "month")
h4 = heatmap(freqtable(data,:cluster,:day)./freqtable(data,:cluster), title = "day")

plot(h1, h2, h3, h4 ,layout=(2,2), legend=false)

Figure 5: scatter plot cluster

It might be a bit confusing initially, but let me take an example that might help you understand. If you take into account cluster 2, it corresponds to the lowest use of the global intensity used. If we go to the heatmap that represents the hours, we will see that the time where this pattern of behavior is most present is at night, which corresponds to the hours we are usually sleeping. I hope this make more sense.

This might give us a slight hint that time frames might be necessary, we'll take this information for featuer engineering at this point later. Now let's start with the next phase.

5. Using EvoTrees for prediction.

For now, we want to predict voltage. I'm not an expert in the field of electricity and consumption, but for a simple exercise, we will use the MLJ library (for Python users it would be equivalent to Scikit-Learn). Due to the amount of data and the algorithm we are going to use, it is not practical to perform training with cross-validation, this will take too much time, so we will prefer to only use a train/test split as a strategy.

let's generate a lag and cut the data in the following way:

data[!, :lag_30] = Array(ShiftedArray(data.Voltage, 30))
replace!(data.lag_30, missing => 0);

And to assign the training and testing, we use the following.

train = copy(filter(x -> x.Date < Date(2010,10,01), data))
test = copy(filter(x -> x.Date >= Date(2010,10,01), data))

Then, we remove some variables that we won't use to train the model, and we save our voltage variable.

select!(train, Not([:Date, :Time, :date_time, :cluster, ]))
select!(test, Not([:Date, :Time, :date_time, :cluster, ]))
y_train = copy(train[!,:Voltage])
y_test = copy(test[!,:Voltage])

Now we are going to apply a cyclical encoder to be able to work with the data, we have several new variables related to time (month, day, hour, among others), and all these variables will be more helpful if we allow extracting their cyclical character, that is why we use a trigonometric transformation

function cyclical_encoder(df::DataFrame, columns::Union{Array, Symbol}, max_val::Union{Array, Int} )
    for (column, max) in zip(columns, max_val)

        df[:, Symbol(string(column) * "_sin")] = sin.(2*pi*df[:, column]/max)
        df[:, Symbol(string(column) * "_cos")] = cos.(2*pi*df[:, column]/max)
    end
    return df
end

Finally, we can apply this new function to our dataset.

columns_selected = [:day, :year, :month, :hour, :minute, :dayofweek]
max_val = [31, 2010, 12, 23, 59, 7]
train_cyclical = cyclical_encoder(train, columns_selected, max_val)
test_cyclical = cyclical_encoder(test, columns_selected, max_val)

And finally, let's train the model.

EvoTreeRegressor = @load EvoTreeRegressor pkg=EvoTrees verbosity=0
etr_start = EvoTreeRegressor(max_depth =15)

machreg = machine(etr_start, train_cyclical[!,14:end], y_train);
fit!(machreg);


pred_etr_train = MLJ.predict(machreg, train_cyclical[!,14:end]);
rms_score_train = rms(pred_etr_train, y_train)
println("The rms in train is $rms_score_train")

pred_etr = MLJ.predict(machreg, test_cyclical[!,14:end]);
rms_score = rms(pred_etr, y_test)
println("The rms in test is $rms_score")

This is our result: * The rms in train is 2.5364451392238085 * The rms in test is 3.438565163838837

In this section, we plot the residual left by our model, and here we can detect some signs of overfitting, considering that our model has a much better score in the training dataset than in the test dataset. On the other hand, the plots are showing us that our model has biases in its predictions, it is not being able to recognize trends.

Figure 6: prediction

Finally, we can see how the predictions compare to the test data.

Figure 7: pred-vs-real

As we have confirmed earlier, the prediction does not seem to have been able to determine the magnitudes of the voltage in the testing of the dataset. Despite the fact that our variable is fairly stable over time, the model was trained with different parameters, but ultimately none of the options managed to show a significant improvement.

6. Conclusions

With this small exercise, we only tried to test that GBM, while a powerful tool and popular in places like Kaggle, requires a certain level of expertise both in the model and in the use case to achieve good performance. A naive approach may not generate results that satisfy the users. This, on one hand, requires:

Understanding how to perform feature engineering for a time series, such as obtaining the decomposition of the time series. This can help capture trends that cannot always be obtained solely with the time horizon.
Applying smoothing strategies like moving averages could help recognize the underlying pattern, but then you will need to estimate that moving average into the future.

Overall, time series analysis requires a deep understanding of the data, proper preprocessing techniques, feature engineering, and selecting appropriate models that can capture the specific patterns and dynamics of the data.

¿Deseas Unirte a la Iglesia de Emacs?

Sat, 23 Nov 2024 00:00:00 +0900

Probablemente si has leído esto desde hace un tiempo, has notado de que usado consistentemente Vi para todo tipo de edición de textos, desde mi blog hasta cuando me toca picar un poco de código, y la verdad es que me gusta harto vim, a tal punto que busqué formas de extender sus bindings en otros programas e incluso en mi administrador de ventanas Sway y dwm. Pero por una curiosidad que venía creciendo decidí probar otro programa que a menudo se le tiende a comparar: Emacs

1. Emacs: El editor de texto programable

Emacs es un editor credo por el gigachad Richard Stallman por allá en 1984, es un software totalmente libre, esta presente en todas las principales plataformas (Windows, MacOS, GNU/Linux), y se caracteriza por ser también un interprete de ELisp (Emacs Lisp) que es basicamente un lenguage de programación derivado de la familia de Lisp, esto lo hace totalmente "hackeable" para cualquiera que quiera abrirse a editar sus configuraciones, ese alto nivel de customización ha logrado que haya una comunidad que este desarrollando librerías bastante convenientes para Emacs

Hay varias cosas enormemente destacables sobre Emacs, pero tal vez una de las cosas que la comunidad más se enorgullece, es que puedes usarlo para un montón de cosas: escribir notas, programar, organizar una agenda, administrar proyectos, leer libros, escuchar música, leer las noticias a través de un RSS, chatear en IRC, jugar Tetris, etc. Todo esto bajo el techo que la iglesia de Emacs puede proveer. Es por esto que algunos tienden a asociar a Emacs como un sistema operativo

Figure 1: emacs

2. Vim vs Emacs

Por alguna razón las comunidades de programadores les gusta comparar estos editores, aunque a mi parecer hay demasiadas diferencias, que uno puede preguntarse si los casos de uso son completamente los mismos, la comparación entre vim e Emacs es como el buen cuchillo y la cocina, en ese sentido mis razones para probar Emacs:

Lisp: Desde hace un tiempo tenía un interés en aprender algún lenguage de la familia de Lisp, es un lenguage imperativo/funcional, altamente flexible, de hecho Scheme otro de los lenguajes Lisperos. era usado como introducción a CS en el MIT, y uno de los libros más cabezones fueron usados como base de enseñanza, Structure and Interpretation of Computer Programs alias SICP, ha sido altamente influyente para este tema. Entonces ciertamente vale la pena darle una vuelta. Un código en Elisp se ve algo como esto:

(setq eshell-prompt-function
  (lambda nil
    (concat
     (eshell/pwd)
     " $ ")))

como puedes notar los paréntesis son una parte previo a la definición de la función y al final, lo hace un tipo de lenguage de programación bastante peculiar en estos tiempos. No es lo único especial de este tipo de lenguajes y hay muchas opiniones al respecto, pero si estás acostumbrado a los lenguajes tipo Python, C o Java, entonces Lisp te va a proveer otro tipo de paradigmas que podrás contrastar.

Ambiente productivo: Usualmente cuando quieres hacer un ambiente productivo en vim, tienes que usar Tmux como multiplexer, y luego concatenar otros programass basados en termimnales, usarás Newsboat para noticias, Ncmpcpp + Mpd para música, Mutt para correos electrónicos, Zathura para leer PDFs, etc. No es un mal workflow, pero tiene sus limitaciones, por ejemplo, si quieres leer blogs que tienen imagenes, Newsboat estará limitado, también tendrás que setear los shortcuts para estos programas para que sean afines a Vim, o los colores de cada software en caso de que te importe lo estético, siento que en general es más simple de abordar esto en Emacs.
Org Mode: Probablemente si hay alguien que me pregunte si vale la pena usar Emacs a pesar de que no programe, probablemente le diré que si, solo por una cosa: Org Mode. Creo que todos independientes del tipo de trabajo o estudio que hacemos, tenemos algunas necesidades de tomar notas, agendar eventos, administrar proyectos con tareas y deadlines, entre otros. Todo eso lo puedes hacer en Emacs con Org Mode, la herramienta es altamente customimizable y es parte integra del ecosistema de Emacs, y creo que no hay ningún usuario de Emacs que no quiera aprovechar los beneficios de este ecosistema.

3. Algunas impresiones no tan positivas de Emacs.

Por sobre todo Emacs es un gran software, este artículo lo he estado escribiendo en Emacs, además de otras cosas, y ha sido divertido por sobre todo, pero algunas cosas que vas a tener que considerar para empezar:

Poco estético al inicio: Si instalas Emacs y vas desde el principio, verás una introducción con el logo bastante feo, además de unos links a tutoriales, etc. No lo encontrarás atractivo para nada, dependerá de ti, ir customizandolo para que quede un poco más agradable visualmente
Nuevos atajos de teclado que aprender: Si vienes de vim, probablemente ya estas acostumbrado al sistema modal de edición, esa memoria muscular tendrá que enfrentar un nuevo sistema de edición y navegación que no tiene mucha relación con Vi, verás que tendrás que usar la tecla Ctrl y Alt para llamar a distintos modos y ejecutar comandos acordes, puede que cueste al principio, pero también tiene sus fortalezas. Además existe una librería para emular los atajos de teclado de vim, llamado evil mode, lo cual puede ayudar a que inicies en Emacs, aunque yo no lo recomiendo, en mi opinión vale la pena que empieces con los atajos originales hasta que logres tener un entendimiento de ellos, y después puedes decidir usar evil mode.
Tiempo mayor en start ups: vim es rápido, Emacs en esencia se toma unos segundos dependiendo de la cantidad de plugins y otras customizaciones que le hayas hecho, lo cual no lo es tanto, pero por lo mismo, preferirás tener una sesión abierta todo el tiemmpo antes que estar abriendo y cerrandolo constantemente, no es algo tan grave en mi opinión, pero puede que haya gente que no le agrade mucho.

4. ¿Qué hay de Doom Emacs, Space Emacs y otros amigos?

Estas distribuciones de Emacs son basicamente Emacs con ciertas opiniones de plugins y shortcuts ya establecidos, no tengo experiencia con esas distribuciones, pero tampoco lo recomiendo mucho, a mi parecer es mejor de que vayas creando tu propio config desde 0 y lo vayas adaptando poco a poco a lo que realmente necesitas, eso también te hará más consciente de lo que hace tu sistema.

5. ¿Cómo hago entonces para partir?

Te recomiendo instalar la distribución básica de Emacs, luego:

Lee el tutorial unas cuantas veces y acostumbrate a usar esos atajos de teclado, por unos cuantos días.
También puedes leer el manual que hizo GNU para Emacs, que tiene un cierto material que complementa al tutorial.
Aprender a navegar dentro de tu directorio con dired, abrir archivos, editarlos y guardarlos.
Entender y quedar cómodo con lo de abrir y cerrar ventanas, operar con los buffers
Explorar otras funcionalidades construídas internamente en Emacs como navegar en la web con eww, navegar pdfs, etc.
Comienza a editar tu archivo init.el para agregar nuevas customizaciones a tu Emacs, como por ejemplo el tamaño de la letra, esquema de colores, etc
Explora Org Mode
Sigue disfrutando.

La verdad es que puede que mi experiencia con Vim y Neovim me hayan dado cierta experiencia de como usar Emacs, eso no quiere decir que no he quedado a veces estancado con algún asunto, pero dentro de todo el proceso ha sido entrete y la verdad me llevado una grata experiencia usando este software. Desde que empecé con el tutorial hasta ahora me ha tomado una semana y media mas o menos de jugar en mi tiempo libre, así que demás que cualquiera puede usarlo con cierta proficiencia en un par de semanas a un mes probablemente.