Hacker News new | past | comments | ask | show | jobs | submit login

> doesn't make a lot of sense (besides privacy...)

Privacy is worth very much though.






What privacy benefit do you get running this locally vs renting a baremetal GPU and running it there?

Wouldn't that be much more cost-effective?

Especially when you inevitably want to run a better / different model in the near future that would benefit from different hardware?

You can get similar Tok/sec on a single RTX 4090 - which you can rent for <$1/hr.


Definitely but when you can run this in places like Azure with tight contracts it makes little sense except for the ultra paranoid.

Considering the power of three letter agencies in the USA and the complete unhingedness of the new administration, I would not trust anything to a contract.

Sure I am certain there is a possibility but unless you have airgapped your local instance and locked down your local network securely it does not really matter.

It’s cool to run things locally and it will get better as time goes on but for most use cases I don’t find it worth it. Everyone is different and folks that enjoy the idea of local network secure can run it locally.


Even a badly operated on-prem system has the advantage that if someone breaks in, they are taking a risk of getting caught. Whereas with Azure the TLAs could just hoover up everything there without high risk of customers finding out (assuming they can gag MS). Given the reporting about NSA's "collect everything" modus operandi this doesn't seem very far fetched.

hmm do we still have to pretend that this is some sort of conspiracy theory? really? after snowden? it doesn't "seem very far fetched", its a fact

It's less "possibility" and more "certainty."

can we even trust the hardware?

The hardware can be airgapped.

Well you don't need to worry unless you are already on the list.

These days getting on a list may require as little as "is trans" or "has immigrant parents."

or "your competitor donated to Musk/Trump campaign"

no, just being speculative... about that.

[flagged]


What does that even mean? Shame these newer account post such low intelligence reaction replies.

For most use cases, you can consider a GCP/AWS/Azure secure.


They are eluding to it not being secure against state actors. The distrust in government isn’t novel to this discussion so it should come as no surprise on HN. There is also a general fear of censorship which should be held more toward the base model owners and not toward cloud providers. I still think doing this in the cloud makes more sense initially but I see the appeal for a home model that is decoupled from the wider ecosystem.



Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: