Artificial Intelligence chatterbot and NLP (Natural Language Processing)

[Like Ra]

2. Torrent for LLaMA models appeared on 4chan almost instantly after downloads were given out to researchers

Any links?

I had to use a 70GB ram buffer

Is it a requirement? Or "a nice to have"? Can memory compression help (e.g. zram-config)? Are the requirements listed anywhere?

24GB of vram

4090 ... 🤤 💸

[dhf7b8g]

Any links?

This is the original magnet from 4chan. magnet:?xt=urn:btih:b8287ebfa04f879b048d4d4404108cf3e8014352&dn=LLaMA&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce

There was a second magnet created and included in a pull request to the official LLaMA repo 

Is it a requirement? Or "a nice to have"? Can memory compression help (e.g. zram-config)? Are the requirements listed anywhere?

It's more of a nice to have. You can load from disk/swap in the webui I linked, but it will be much slower than having a model in VRAM, 8bit quant doesn't work when split across GPU+RAM/swap either.
Memory compression might help, but personally I haven't tried it.

There are no official requirements, as they expect you to have a handful of A100s if you are getting this model. I would recommend filesize+2GB as the amount of VRAM needed for "good" speeds as VRAM is a much more limiting factor than GPU power, but you can even run it from disk and using a CPU (as long as you don't care about speed)

[Lancer]

Been searching the internet for one of Googles pet projects, specifically the one detailed in arxiv and their recent palm publication called GLaM, mostly since it claims to have a 50 percent computational decrease compared to chat gpt3 by using multiple feed forward networks at the same time.

Ps does anyone know of a transformer attention mechanism that isn't made in python?

[Like Ra]

transformer attention mechanism that isn't made in python?

Performance-wise? Isn't what Torch (e.g. pytorch, LUA torch) is for? All computational intensive things are written in C.

[egregious]

transformer attention mechanism that isn't made in python?

Performance-wise? Isn't what Torch (e.g. pytorch, LUA torch) is for? All computational intensive things are written in C.

I thought it was only use greybeards that wrote C these days.
Everyone else seems to be script kiddies or glorified integration engineers (as that is what most of the jobs I see on offer are).

An awful lot of knowledge on how to write optimised code that runs on lower spec (ie less power hungry and cheaper) devices is being lost as we retire

[Like Ra]

I thought it was only use greybeards that wrote C these days.

https://github.com/pytorch/pytorch
C++ 45.8%
C 2.4%

https://github.com/torch/torch7
C 65.0%
C++ 0.5%

😊

[Lancer]

transformer attention mechanism that isn't made in python?

Performance-wise? Isn't what Torch (e.g. pytorch, LUA torch) is for? All computational intensive things are written in C.

I am a java programmer and have okay ability to read and call c libraries but I cannot for the life of me find one that doesn't have pythons sticky fingers laced in. Using java to call python to call C is wasteful, very slow and requires strange project managers like maven which are the bane of my existence. In theory transformers aren't even supposed to be that hard so I am now attempting to code one from scratch but it is still a pain in the butt.

[Lancer]

transformer attention mechanism that isn't made in python?

Performance-wise? Isn't what Torch (e.g. pytorch, LUA torch) is for? All computational intensive things are written in C.

I thought it was only use greybeards that wrote C these days.
Everyone else seems to be script kiddies or glorified integration engineers (as that is what most of the jobs I see on offer are).

An awful lot of knowledge on how to write optimised code that runs on lower spec (ie less power hungry and cheaper) devices is being lost as we retire

Yeah coding in C seems to be an exercise in masochism and while I am not afraid to get my hands a bit dirty with coding I don't really feel too constrained by hardware limits. Even your average raspberry pi 4 is now quad core which means 1 core gets used to run the pi and 3 are up for grabs using multithreaded Java and with the innovations in virtual threading performance is less of a concern.

[Zooy]

transformer attention mechanism that isn't made in python?

Performance-wise? Isn't what Torch (e.g. pytorch, LUA torch) is for? All computational intensive things are written in C.

I thought it was only use greybeards that wrote C these days.
Everyone else seems to be script kiddies or glorified integration engineers (as that is what most of the jobs I see on offer are).

An awful lot of knowledge on how to write optimised code that runs on lower spec (ie less power hungry and cheaper) devices is being lost as we retire

My life's work is an application, written in C, with some added libraries written in C++, but those were written by others.
This application is used on very powerful computers to get the most out of them. The alternative for the users would be to buy several times as many computers or wait several years until the computers are more powerful.
But I am also retired now and I am trying to pass the proper attitude on to a number of the people who rely on this application.

One of the problems is that the new generation expects to pick up ready made modules without knowing what is in them, and other people who are so self-confident that they advertise their often not very high quality products. I have seen a publication where the authors were boasting that they could calculate some mathematical entity in 'only a few hours' on a decent computer, after which in the next issue somebody commented that he calculated the same quantity in a leisurely 5 minutes by hand and noticed the following errors in their answer......... Just imagine that you pick up such a program.

[Like Ra]

Using java to call python to call C is wasteful

Totally agree! Just dump Java and everything is Python-simple and C-fast! 😬

Even your average raspberry pi 4 is now quad core

Yep, what makes it very tempting to forget efficiency.

Actually, the most efficient scientific libraries are still written in Fortran.

are up for grabs using multithreaded Java

Provided the code is written with multithreading in mind.

Speaking about neural nets - Java is not needed here. All you need is C(++) and Cuda for the low-level math calculations and Python for the interface. Who cares, that Python is slow, it's not used for calculations anyway. And if you need more speed from Python or Lua there is Cython (yes, C again) and Luajit (yes, C again).

And ... JVM/JRE and native Java libraries are also written in C/C++ 😆