Does this mean that if some hacker found out what my
website was made
of, he could easily find his way in? Am I making a site that is more
easier to hack, as all source is free-ly avaiable for them to find
security holes?
Don't believe in security through obscurity. Open Source may mean people
find holes, but if that happens they are quickly fixed - and then they
are no longer there. Much better than if they are there, only not (yet)
known.